Post by Sportfreunde » Wed Mar 04, 2020 6:41 pm

Hello everybody

At the beginning I tried SEO modules (free). But Google always had something to complain about. Error in the robots file or sitemap is not readable. A user told me that it is most likely due to the extension. I have now uninstalled this extension. Now there are more side activities to see, but they are not visitors.

I wonder how to set up htaccess, robots and sitemap properly. How do I best configure the files? I created sitemap via extensions feeds but also Google says that the sitemap cannot be read.

****************************

Code: Select all

Options +FollowSymlinks

# Prevent Directoy listing
Options -Indexes

# Prevent Direct Access to files
<FilesMatch "(?i)((\.tpl|.twig|\.ini|\.log|(?<!robots)\.txt))">
 Require all denied
</FilesMatch>

# SEO URL Settings
RewriteEngine On

# If your opencart installation does not run on the main web folder make sure you folder it does run in ie. / becomes /shop/

RewriteBase /

###############################################
## force https                               ##
## - ATTENTION: this force also admin pages! ##
###############################################

## non Cloud Systems 
#RewriteCond %{HTTPS} off
#RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

## Cloud Systems 
## e.g. CloudFlare, AWS Elastic Load Balancing, Heroku, 
## OpenShift or any other Cloud/PaaS
# If we receive a forwarded http request from a proxy...
RewriteCond %{HTTP:X-Forwarded-Proto} =http [OR]

# ...or just a plain old http request directly from the client
RewriteCond %{HTTP:X-Forwarded-Proto} =""
RewriteCond %{HTTPS} !=on
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

## force non-www
RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
RewriteRule ^(.*)$ https://%1/$1 [R=301,L]

RewriteRule ^sitemap.xml$ index.php?route=extension/feed/simple_google_sitemap [L] 
RewriteRule ^googlebase.xml$ index.php?route=extension/feed/google_base [L]
RewriteRule ^system/storage/(.*) index.php?route=error/not_found [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !.*\.(ico|gif|jpg|jpeg|png|js|css)
RewriteRule ^([^?]*) index.php?_route_=$1 [L,QSA]

# BEGIN cPanel-generated php ini directives, do not edit
# Manual editing of this file may result in unexpected behavior.
# To make changes to this file, use the cPanel MultiPHP INI Editor (Home >> Software >> MultiPHP INI Editor)
# For more information, read our documentation (https://go.cpanel.net/EA4ModifyINI)
<IfModule php7_module>
   php_value default_charset "UTF-8"
   php_value memory_limit 64M
   php_value max_execution_time 360
   php_value upload_max_filesize 99M
   php_value mysql.connect_timeout 20
   php_flag session.auto_start Off
   php_flag session.use_only_cookies On
   php_flag session.use_cookies On
   php_flag session.use_trans_sid Off
   php_value session.cookie_httponly "1"
   php_value session.gc_maxlifetime 3600
   php_flag display_errors On
   php_value max_input_time 60
   php_value max_input_vars 4000
   php_value post_max_size 8M
   php_value session.save_path "/var/cpanel/php/sessions/ea-php73"
   php_flag zlib.output_compression On
</IfModule>
<IfModule lsapi_module>
   php_value default_charset "UTF-8"
   php_value memory_limit 64M
   php_value max_execution_time 360
   php_value upload_max_filesize 99M
   php_value mysql.connect_timeout 20
   php_flag session.auto_start Off
   php_flag session.use_only_cookies On
   php_flag session.use_cookies On
   php_flag session.use_trans_sid Off
   php_value session.cookie_httponly "1"
   php_value session.gc_maxlifetime 3600
   php_flag display_errors On
   php_value max_input_time 60
   php_value max_input_vars 4000
   php_value post_max_size 8M
   php_value session.save_path "/var/cpanel/php/sessions/ea-php73"
   php_flag zlib.output_compression On
</IfModule>
# END cPanel-generated php ini directives, do not edit

# php -- BEGIN cPanel-generated handler, do not edit
# Set the “ea-php73” package as the default “PHP” programming language.
<IfModule mime_module>
  AddHandler application/x-httpd-ea-php73 .php .php7 .phtml
</IfModule>
# php -- END cPanel-generated handler, do not edit
Can someone help me?

*******************************************

Is htaccess safe for the directories or is something still missing?

can someone help me?
Last edited by straightlight on Thu Mar 05, 2020 1:35 am, edited 1 time in total.
Reason: Please add the code tags.

New member

Posts

Joined
Sat Nov 30, 2019 4:46 am

Post by Quino » Wed Mar 04, 2020 9:22 pm

well, why not state what the problem is?
The rewrite rule here for sitemap is nothing more than to enable you to enter sitemap.xml instead of index.php?......bla bla bla.
There is no need for that as you can just submit the normal link to google, they don't care if it doesn't look cute.
If google cannot read your sitemap then you may have problems with your "simple_google_sitemap" controller, check if you can read that in your browser.

Robot.txt has no relation here, there you would just disallow any url with checkout or account or parameters in them but you can much better set x-robots headers and robots meta tags dynamically.

SEO urls require this part:

Code: Select all

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !.*\.(ico|gif|jpg|jpeg|png|js|css)
RewriteRule ^([^?]*) index.php?_route_=$1 [L,QSA]
which basically means that is the requested url is not an existing file or directory and it does not contain any of those file extensions, just hand the url over to the index.php script as a parameter and let php figure out what the page should be to be returned. That part is then done via the seo_url.php script.

New member

Posts

Joined
Fri Feb 28, 2020 1:19 am

Post by Sportfreunde » Thu Mar 05, 2020 1:19 am

Good evening

Thank you for your reply. Excuse my ignorance. I don't have that much experience in this area. I uninstalled the SEO extension. The problem is that something can be wrong between htaccess sitemap and robots.txt. I have a lot of hits on the site but they are not customers. Google doesn't recognize sitemap and robots.txt. Goggle says is flawed. Sitemap is not readable by Google.

What would you recommend me how the htaccess should look so that it runs properly. I should have read the Protect directories too.

Thanks for your help

New member

Posts

Joined
Sat Nov 30, 2019 4:46 am

Post by sw!tch » Thu Mar 05, 2020 2:32 am

Link to your site?

Full Stack Web Developer :: Send a PM for Custom Work.
Backup and learn how to recover before you make any changes!


Active Member

Posts

Joined
Sat Apr 28, 2012 2:32 pm

Post by Quino » Thu Mar 05, 2020 10:50 am

First check if you get a result by requesting your sitemap in your browser by using

Code: Select all

https://YOUR-DOMAIN/index.php?route=extension/feed/simple_google_sitemap
for robots.txt I use a very simple one for OC

Code: Select all

Sitemap: MY-DOMAIN/index.php?route=extension/feed/google_sitemap

User-agent: *
Disallow: /*route=account/*
Disallow: /*route=affiliate/*
Disallow: /*route=checkout/*
Disallow: /*route=error/*
Disallow: /*route=extension/module/*

Host: MY-DOMAIN
But I do set robots headers and meta tags per page by adding this in the catalog/controller/common/header.php

Code: Select all

		// index and follow headers
		$data['robots'] = false;
		if (isset($_SERVER['QUERY_STRING']) && 
			(
			stristr($_SERVER['QUERY_STRING'],'search=') or
			stristr($_SERVER['QUERY_STRING'],'order=') or
			stristr($_SERVER['QUERY_STRING'],'limit=') or
			stristr($_SERVER['QUERY_STRING'],'start=') or
			stristr($_SERVER['QUERY_STRING'],'page=') or
			stristr($_SERVER['QUERY_STRING'],'sort=')
			)
			) {
			header("X-Robots-Tag: noindex, follow", true);
			$data['robots'] = '<meta name="robots" content="noindex, follow">';
		} elseif (isset($this->request->get['route']) && 
			(
			stristr($this->request->get['route'],'checkout') or
			stristr($this->request->get['route'],'account') or
			stristr($this->request->get['route'],'module') or
			stristr($this->request->get['route'],'error')
			)
			) {
			header("X-Robots-Tag: noindex, nofollow", true);
			$data['robots'] = '<meta name="robots" content="noindex, nofollow">';
		} elseif (isset($this->request->get['route']) && 
			(
			stristr($this->request->get['route'],'FUTURE_PLACEHOLDER')
			)
			) {
			header("X-Robots-Tag: index, nofollow", true);
			$data['robots'] = '<meta name="robots" content="index, nofollow">';
		} else {
			header("X-Robots-Tag: index, follow", true);
			$data['robots'] = '<meta name="robots" content="index, follow">';
		}
and add

Code: Select all

{% if robots %}
{{ robots }}
{% endif %}
to the header view to set the meta tag there

New member

Posts

Joined
Fri Feb 28, 2020 1:19 am

Post by paulfeakins » Thu Mar 05, 2020 7:27 pm

Sportfreunde wrote:
Wed Mar 04, 2020 6:41 pm
Google always had something to complain about.
Get that list and fix them one by one.

UK OpenCart Hosting | OpenCart Audits | OpenCart Support - please email info@antropy.co.uk


User avatar
Guru Member
Online

Posts

Joined
Mon Aug 22, 2011 11:01 pm
Location - London Gatwick, United Kingdom

Post by Sportfreunde » Fri Mar 06, 2020 9:00 pm

Quino wrote:
Thu Mar 05, 2020 10:50 am
First check if you get a result by requesting your sitemap in your browser by using

Code: Select all

https://YOUR-DOMAIN/index.php?route=extension/feed/simple_google_sitemap 
Does not work / XML processing error: Undefined entity / Row No. 565, column 35:

Code: Select all

This is the Sitmap
<?php
class ControllerExtensionFeedGoogleSitemap extends Controller {
	private $error = array();

	public function index() {
		$this->load->language('extension/feed/google_sitemap');

		$this->document->setTitle($this->language->get('heading_title'));

		$this->load->model('setting/setting');

		if (($this->request->server['REQUEST_METHOD'] == 'POST') && $this->validate()) {
			$this->model_setting_setting->editSetting('feed_google_sitemap', $this->request->post);

			$this->session->data['success'] = $this->language->get('text_success');

			$this->response->redirect($this->url->link('marketplace/extension', 'user_token=' . $this->session->data['user_token'] . '&type=feed', true));
		}

		if (isset($this->error['warning'])) {
			$data['error_warning'] = $this->error['warning'];
		} else {
			$data['error_warning'] = '';
		}

		$data['breadcrumbs'] = array();

		$data['breadcrumbs'][] = array(
			'text' => $this->language->get('text_home'),
			'href' => $this->url->link('common/dashboard', 'user_token=' . $this->session->data['user_token'], true)
		);

		$data['breadcrumbs'][] = array(
			'text' => $this->language->get('text_extension'),
			'href' => $this->url->link('marketplace/extension', 'user_token=' . $this->session->data['user_token'] . '&type=feed', true)
		);

		$data['breadcrumbs'][] = array(
			'text' => $this->language->get('heading_title'),
			'href' => $this->url->link('extension/feed/google_sitemap', 'user_token=' . $this->session->data['user_token'], true)
		);

		$data['action'] = $this->url->link('extension/feed/google_sitemap', 'user_token=' . $this->session->data['user_token'], true);

		$data['cancel'] = $this->url->link('marketplace/extension', 'user_token=' . $this->session->data['user_token'] . '&type=feed', true);

		if (isset($this->request->post['feed_google_sitemap_status'])) {
			$data['feed_google_sitemap_status'] = $this->request->post['feed_google_sitemap_status'];
		} else {
			$data['feed_google_sitemap_status'] = $this->config->get('feed_google_sitemap_status');
		}

		$data['data_feed'] = HTTP_CATALOG . 'index.php?route=extension/feed/google_sitemap';

		$data['header'] = $this->load->controller('common/header');
		$data['column_left'] = $this->load->controller('common/column_left');
		$data['footer'] = $this->load->controller('common/footer');

		$this->response->setOutput($this->load->view('extension/feed/google_sitemap', $data));
	}

	protected function validate() {
		if (!$this->user->hasPermission('modify', 'extension/feed/google_sitemap')) {
			$this->error['warning'] = $this->language->get('error_permission');
		}

		return !$this->error;
	}
}
for robots.txt I use a very simple one for OC

Code: Select all

Sitemap: MY-DOMAIN/index.php?route=extension/feed/google_sitemap

User-agent: *
Disallow: /*route=account/*
Disallow: /*route=affiliate/*
Disallow: /*route=checkout/*
Disallow: /*route=error/*
Disallow: /*route=extension/module/*

Host: MY-DOMAIN

I added this in the Robots
But I do set robots headers and meta tags per page by adding this in the catalog/controller/common/header.php

Code: Select all

		// index and follow headers
		$data['robots'] = false;
		if (isset($_SERVER['QUERY_STRING']) && 
			(
			stristr($_SERVER['QUERY_STRING'],'search=') or
			stristr($_SERVER['QUERY_STRING'],'order=') or
			stristr($_SERVER['QUERY_STRING'],'limit=') or
			stristr($_SERVER['QUERY_STRING'],'start=') or
			stristr($_SERVER['QUERY_STRING'],'page=') or
			stristr($_SERVER['QUERY_STRING'],'sort=')
			)
			) {
			header("X-Robots-Tag: noindex, follow", true);
			$data['robots'] = '<meta name="robots" content="noindex, follow">';
		} elseif (isset($this->request->get['route']) && 
			(
			stristr($this->request->get['route'],'checkout') or
			stristr($this->request->get['route'],'account') or
			stristr($this->request->get['route'],'module') or
			stristr($this->request->get['route'],'error')
			)
			) {
			header("X-Robots-Tag: noindex, nofollow", true);
			$data['robots'] = '<meta name="robots" content="noindex, nofollow">';
		} elseif (isset($this->request->get['route']) && 
			(
			stristr($this->request->get['route'],'FUTURE_PLACEHOLDER')
			)
			) {
			header("X-Robots-Tag: index, nofollow", true);
			$data['robots'] = '<meta name="robots" content="index, nofollow">';
		} else {
			header("X-Robots-Tag: index, follow", true);
			$data['robots'] = '<meta name="robots" content="index, follow">';
		}
		
		I added this in the header. But I get an error message. See attachement
and add

Code: Select all

{% if robots %}
{{ robots }}
{% endif %}

I do not understand where I have to add that. Also in the header file and where?
to the header view to set the meta tag there

Attachments

header.jpg

header.jpg (138.46 KiB) Viewed 2866 times


New member

Posts

Joined
Sat Nov 30, 2019 4:46 am

Post by Quino » Fri Mar 06, 2020 9:44 pm

that is your admin sitemap controller, not the one google will trigger, the catalog one.

It would be helpfull if you would put the code BEFORE the view is called, not after.
so before

Code: Select all

return $this->load->view('common/header', $data);
Last edited by Quino on Fri Mar 06, 2020 9:51 pm, edited 1 time in total.

New member

Posts

Joined
Fri Feb 28, 2020 1:19 am

Post by Sportfreunde » Fri Mar 06, 2020 9:50 pm

This is the cPanel editor. What can I do with the other questions now?

New member

Posts

Joined
Sat Nov 30, 2019 4:46 am

Post by Quino » Fri Mar 06, 2020 9:52 pm

see edited post

New member

Posts

Joined
Fri Feb 28, 2020 1:19 am

Post by Sportfreunde » Fri Mar 06, 2020 10:41 pm

Where do I insert it? In the sitemap?

In this case, this is the sitemap for Google

Code: Select all

<?php
class ControllerExtensionFeedGoogleSitemap extends Controller {
	public function index() {
		if ($this->config->get('feed_google_sitemap_status')) {
			$output  = '<?xml version="1.0" encoding="UTF-8"?>';
			$output .= '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">';

			$this->load->model('catalog/product');
			$this->load->model('tool/image');

			$products = $this->model_catalog_product->getProducts();

			foreach ($products as $product) {
				if ($product['image']) {
					$output .= '<url>';
					$output .= '  <loc>' . $this->url->link('product/product', 'product_id=' . $product['product_id']) . '</loc>';
					$output .= '  <changefreq>weekly</changefreq>';
					$output .= '  <lastmod>' . date('Y-m-d\TH:i:sP', strtotime($product['date_modified'])) . '</lastmod>';
					$output .= '  <priority>1.0</priority>';
					$output .= '  <image:image>';
					$output .= '  <image:loc>' . $this->model_tool_image->resize($product['image'], $this->config->get('theme_' . $this->config->get('config_theme') . '_image_popup_width'), $this->config->get('theme_' . $this->config->get('config_theme') . '_image_popup_height')) . '</image:loc>';
					$output .= '  <image:caption>' . $product['name'] . '</image:caption>';
					$output .= '  <image:title>' . $product['name'] . '</image:title>';
					$output .= '  </image:image>';
					$output .= '</url>';
				}
			}

			$this->load->model('catalog/category');

			$output .= $this->getCategories(0);

			$this->load->model('catalog/manufacturer');

			$manufacturers = $this->model_catalog_manufacturer->getManufacturers();

			foreach ($manufacturers as $manufacturer) {
				$output .= '<url>';
				$output .= '  <loc>' . $this->url->link('product/manufacturer/info', 'manufacturer_id=' . $manufacturer['manufacturer_id']) . '</loc>';
				$output .= '  <changefreq>weekly</changefreq>';
				$output .= '  <priority>0.7</priority>';
				$output .= '</url>';

				$products = $this->model_catalog_product->getProducts(array('filter_manufacturer_id' => $manufacturer['manufacturer_id']));

				foreach ($products as $product) {
					$output .= '<url>';
					$output .= '  <loc>' . $this->url->link('product/product', 'manufacturer_id=' . $manufacturer['manufacturer_id'] . '&product_id=' . $product['product_id']) . '</loc>';
					$output .= '  <changefreq>weekly</changefreq>';
					$output .= '  <priority>1.0</priority>';
					$output .= '</url>';
				}
			}

			$this->load->model('catalog/information');

			$informations = $this->model_catalog_information->getInformations();

			foreach ($informations as $information) {
				$output .= '<url>';
				$output .= '  <loc>' . $this->url->link('information/information', 'information_id=' . $information['information_id']) . '</loc>';
				$output .= '  <changefreq>weekly</changefreq>';
				$output .= '  <priority>0.5</priority>';
				$output .= '</url>';
			}

			$output .= '</urlset>';

			$this->response->addHeader('Content-Type: application/xml');
			$this->response->setOutput($output);
		}
	}

	protected function getCategories($parent_id, $current_path = '') {
		$output = '';

		$results = $this->model_catalog_category->getCategories($parent_id);

		foreach ($results as $result) {
			if (!$current_path) {
				$new_path = $result['category_id'];
			} else {
				$new_path = $current_path . '_' . $result['category_id'];
			}

			$output .= '<url>';
			$output .= '  <loc>' . $this->url->link('product/category', 'path=' . $new_path) . '</loc>';
			$output .= '  <changefreq>weekly</changefreq>';
			$output .= '  <priority>0.7</priority>';
			$output .= '</url>';

			$products = $this->model_catalog_product->getProducts(array('filter_category_id' => $result['category_id']));

			foreach ($products as $product) {
				$output .= '<url>';
				$output .= '  <loc>' . $this->url->link('product/product', 'path=' . $new_path . '&product_id=' . $product['product_id']) . '</loc>';
				$output .= '  <changefreq>weekly</changefreq>';
				$output .= '  <priority>1.0</priority>';
				$output .= '</url>';
			}

			$output .= $this->getCategories($result['category_id'], $new_path);
		}

		return $output;
	}
}

New member

Posts

Joined
Sat Nov 30, 2019 4:46 am

Post by Quino » Fri Mar 06, 2020 10:58 pm

1) concerning the robots headers and meta tags
no, you already put it in the right file (common/header controller), just on the wrong position.
and this code:

Code: Select all

{% if robots %}
{{ robots }}
{% endif %}
you add to the common/header twig file between the <head>...</head> tags

2) concerning the sitemap
The sitemap controller code you posted is the wrong one, google will try to fetch the sitemap controller from the catalog side, not the admin side.
it is the catalog/controller/extension/feed/simple_google_sitemap.php which gives the error, not the admin/controller/extension/feed/google_sitemap.php

New member

Posts

Joined
Fri Feb 28, 2020 1:19 am

Post by Quino » Fri Mar 06, 2020 11:00 pm

so what happened to "simple_google_sitemap"?

New member

Posts

Joined
Fri Feb 28, 2020 1:19 am

Post by Sportfreunde » Sat Mar 07, 2020 12:24 am

My page was offline after inserting the code for header. I have now uploaded the original header.php again. Page is online again.

Really complicated if you don't have as much idea about the whole thing.

New member

Posts

Joined
Sat Nov 30, 2019 4:46 am

Post by Quino » Sat Mar 07, 2020 12:33 am

because you put the code in the wrong place.

New member

Posts

Joined
Fri Feb 28, 2020 1:19 am

Post by Sportfreunde » Sat Mar 07, 2020 1:16 am

Sorry I don't have that much idea. Can you tell me where or in which line I should use it

New member

Posts

Joined
Sat Nov 30, 2019 4:46 am

Post by Quino » Sat Mar 07, 2020 1:31 am

BEFORE the line:

Code: Select all

return $this->load->view('common/header', $data);
in your editor that was line 80

New member

Posts

Joined
Fri Feb 28, 2020 1:19 am

Post by Sportfreunde » Sat Mar 07, 2020 1:36 am

worked!!! So i tested the next step!

New member

Posts

Joined
Sat Nov 30, 2019 4:46 am

Post by Sportfreunde » Sat Mar 07, 2020 1:44 am

Regarding the sitemap, as you say:

/public_html/catalog/controller/extension/feed/google_sitemap.php

be used. But where do I have to make the change that Google gets the right sitemap?

New member

Posts

Joined
Sat Nov 30, 2019 4:46 am

Post by Quino » Sat Mar 07, 2020 1:47 am

does that sitemap work?
just use your browser and put the url in the address bar, you should see exactly what google would see.

New member

Posts

Joined
Fri Feb 28, 2020 1:19 am
Who is online

Users browsing this forum: Baidu [Spider] and 186 guests