Post by AndreyPopov » Sun Jul 05, 2020 6:13 am

letxobnav wrote:
Sat Jul 04, 2020 9:59 pm
besides, in my initiator it states push/other, in yours just other:
now sets to use cookies

letxobnav wrote:
Sat Jul 04, 2020 9:59 pm
So I still think it is a server/lightspeed issue.
hoster says that size of headers are 16k

but I think response header size is limit to 4k.

how check response header size?

size of response header near 500-600 bytes
now, when I limit links in header to 25 than size of added links are 3200-3400 bytes

600+3400 + next link ~150 bytes exceed size limit of 4k - 404 error?

how check real size limit of response header?
Last edited by AndreyPopov on Sun Jul 05, 2020 8:07 pm, edited 1 time in total.

New member

Posts

Joined
Sat Feb 04, 2017 2:53 am

Post by letxobnav » Sun Jul 05, 2020 7:39 am

I only know of a request line limit, not a response line limit except for maxHttpHeaderSize in apache tomcat, do not know of a setting in nginx.
But hosts can of course limit it, some CDNs like Akamai do, that includes the cookies as well though.

either try sending individual headers per asset or use strlen to count the characters.

or if phones are the issue, I limit the number of products I display on phones which also means less images which means less headers.
Not sure if journal can do that but you could do the same by detecting devices.

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by AndreyPopov » Sun Jul 05, 2020 8:13 pm

letxobnav wrote:
Sun Jul 05, 2020 7:39 am
Not sure if journal can do that but you could do the same by detecting devices.
Journal have separate view:
- desktop
- tablet
- mobile
and also separate settings for each view
also I can separate settings for Customer, Customer Group, Stores.

letxobnav wrote:
Sun Jul 05, 2020 7:39 am
use strlen to count the characters.
already. but I count only added links.
and time to time even I limit to 25 links:

Code: Select all

05-Jul-2020 15:05:37 Europe/Kiev	3489 bytes in header with added links
later try to test on which size become 404 or 503 error.
letxobnav wrote:
Sun Jul 05, 2020 7:39 am
I only know of a request line limit, not a response line limit except for maxHttpHeaderSize in apache tomcat, do not know of a setting in nginx.
But hosts can of course limit it, some CDNs like Akamai do, that includes the cookies as well though.
hoster says headers are 16k, but …… total? or for each?
I ask hoster again.


find in nginx docs:
Syntax: large_client_header_buffers number size;
Default: large_client_header_buffers 4 8k;
Context: http, server

Sets the maximum number and size of buffers used for reading large client request header. A request line cannot exceed the size of one buffer, or the 414 (Request-URI Too Large) error is returned to the client. A request header field cannot exceed the size of one buffer as well, or the 400 (Bad Request) error is returned to the client. Buffers are allocated only on demand. By default, the buffer size is equal to 8K bytes. If after the end of request processing a connection is transitioned into the keep-alive state, these buffers are released.

New member

Posts

Joined
Sat Feb 04, 2017 2:53 am

Post by letxobnav » Sun Jul 05, 2020 8:58 pm

when you talk to your host, ask them is they have a setting/facility which limits the requests per second.
when you push, the client will request many assets in parallel (multiplexed using the same tcp connection) when using http/2, perhaps your host limits that amount and that is why your client gets a 404 or 503.

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by AndreyPopov » Mon Jul 06, 2020 5:03 am

letxobnav wrote:
Sun Jul 05, 2020 8:58 pm
when you talk to your host, ask them is they have a setting/facility which limits the requests per second.
when you push, the client will request many assets in parallel (multiplexed using the same tcp connection) when using http/2, perhaps your host limits that amount and that is why your client gets a 404 or 503.
something wrong in this sentence?

mobile view send less requests than desktop view.

same pages in desktop view can accept in header more links (35-40) than mobile view that give 404 error at 30 links in header.

New member

Posts

Joined
Sat Feb 04, 2017 2:53 am

Post by AndreyPopov » Tue Jul 07, 2020 10:59 pm

some experiments and investigations and I found that:

string "link: ..." cannot be more than 4096 bytes!!!!

make separate "link: ..." for each css, js, font, mp3 and images

for images plan also make separate links for preload and prefetch.

New member

Posts

Joined
Sat Feb 04, 2017 2:53 am

Post by letxobnav » Wed Jul 08, 2020 2:09 am

That might be a server restriction.
saw this from CF, maybe lightspeed has limits too.
https://blog.cloudflare.com/http-2-serv ... nk-header/

But your server still does not show push as request initiator even when preload headers are set and there is no "receiving push" on those assets in the waterfall so I am pretty sure you are actually not pushing anything.

Still. I would suggest you reduce your home page.
You have 306 requests on desktop, 230 on tablet and 138 on phone, 95% of those are images.
Given that the source code is also around 8300 lines of code does not help, even when I only view source my browser times out on that.

Don't try to put your entire store on the home page.

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by AndreyPopov » Wed Jul 08, 2020 2:59 am

letxobnav wrote:
Wed Jul 08, 2020 2:09 am
Still. I would suggest you reduce your home page.
home page is not my main "head pain"

404 and 503 was on other pages.

letxobnav wrote:
Wed Jul 08, 2020 2:09 am
But your server still does not show push as request initiator even when preload headers are set and there is no "receiving push" on those assets in the waterfall so I am pretty sure you are actually not pushing anything.
syntax error on your code that sets relative: your code remove head slash at the begin of file name.
I return head / and see Push / Other


P.S. there are problems with Push Journal's css and js :(

New member

Posts

Joined
Sat Feb 04, 2017 2:53 am

Post by letxobnav » Wed Jul 08, 2020 8:16 am

your code remove head slash at the begin of file name
Then set the subdir correctly.

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by AndreyPopov » Wed Jul 08, 2020 5:55 pm

letxobnav wrote:
Wed Jul 08, 2020 8:16 am
your code remove head slash at the begin of file name
Then set the subdir correctly.
already done.


now Push, but some problem with admin side.
when css and/or js Push work in admin side generated some non-exist error like:

Code: Select all

www.priazha-shop.com/view/assets/css/bootstrap-multiselect.min.css	
www.priazha-shop.com/view/assets/css/ocmodify.css	
www.priazha-shop.com/view/javascript/d_bootstrap_switch/css/bootstrap-switch.css	
www.priazha-shop.com/view/javascript/d_bootstrap_switch/css/bootstrap-switch.css	
www.priazha-shop.com/view/javascript/d_bootstrap_switch/css/bootstrap-switch.css	
www.priazha-shop.com/view/javascript/font-awesome/css/font-awesome.min.css
frontend work good.

errors only while work in admin.


and when trying enter to Journal Control Panel then first error
Image
and after 3-5 seconds Journal Control Panel loaded :(

New member

Posts

Joined
Sat Feb 04, 2017 2:53 am

Post by AndreyPopov » Mon Jul 13, 2020 9:25 pm

main conclusions about Server Push

1. link string in header must be less than 4096 bytes.
when exceed this limit then 404 or 503 error
that's why use separate link for each or control size

2. for Push in link accepted only filename with path like /catalog/view...../style.css
filename like url https://www.domain.com/page not pushed

New member

Posts

Joined
Sat Feb 04, 2017 2:53 am

Post by letxobnav » Mon Jul 13, 2020 10:46 pm

errors only while work in admin.
well, turn push off for admin, no need there.
use:

Code: Select all

if ($_SERVER['SERVER_PROTOCOL'] == 'HTTP/2.0' && !strstr($_SERVER['REQUEST_URI'],'admin/')) {
	[push code]
}
1. link string in header must be less than 4096 bytes.
when exceed this limit then 404 or 503 error
that's why use separate link for each or control size
Don't have that issue.
2. for Push in link accepted only filename with path like /catalog/view...../style.css
filename like url https://www.domain.com/page not pushed
of course, you push static assets, not paths.

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by AndreyPopov » Tue Jul 14, 2020 6:36 pm

letxobnav wrote:
Mon Jul 13, 2020 10:46 pm
errors only while work in admin.
well, turn push off for admin, no need there.
use:

Code: Select all

if ($_SERVER['SERVER_PROTOCOL'] == 'HTTP/2.0' && !strstr($_SERVER['REQUEST_URI'],'admin/')) {
	[push code]
}
thanks. I will try.

P.S. really work!!!! GREAT thanks!!!!!!



letxobnav wrote:
Mon Jul 13, 2020 10:46 pm
1. link string in header must be less than 4096 bytes.
when exceed this limit then 404 or 503 error
that's why use separate link for each or control size
Don't have that issue.
may be this is LightSpeed server issue.
but when I limit to 4096 bytes my problems with 404 and 503 errors gone.

letxobnav wrote:
Mon Jul 13, 2020 10:46 pm
2. for Push in link accepted only filename with path like /catalog/view...../style.css
filename like url https://www.domain.com/page not pushed
of course, you push static assets, not paths.
but initial in your code:
when $relative = true then head slash absent and push not work
when $relative = false then always path with https://www.domain.com/ and push not work
initial in your code push cannot work in any way :(

that's why I set $relative = false and add

Code: Select all

$file = str_replace(HTTPS_SERVER,'/',$file);
before each

Code: Select all

$headersImage[] = '<'.$file.'>;rel='.$method_img.';as=image';

New member

Posts

Joined
Sat Feb 04, 2017 2:53 am

Post by AndreyPopov » Tue Jul 14, 2020 11:20 pm

next strange:

only 5 (five) images pushed.
why?

New member

Posts

Joined
Sat Feb 04, 2017 2:53 am

Post by AndreyPopov » Fri Jul 17, 2020 9:37 pm

my host/server/site configuration not Push more than 5-6 images.

that's why no matter detect all images on page and add they to header.

another problem with Journal:
on desktop view and mobile view Journal use images with different resolutions

for example:
main product: desktop 550x550, mobile iPhone 1100x1100
same products: desktop 190x190, mobile iPhone 380x380
images in filter: desktop 37x37, mobile iPhone 74x74

and main in this, that:
- detection algorithm not find 1100x1100, 380x380, 74x74 etc.
- add and try push on mobile 550x550, 190x190, 37x37

:(

that's why I disable detection algorithm for images and add images manually.

New member

Posts

Joined
Sat Feb 04, 2017 2:53 am

Post by letxobnav » Fri Jul 17, 2020 11:39 pm

The code detects asset files in the html output you are about to send and sends push headers for those assets ahead of the html.
It does not care whether that html output is for mobiles or desktops and it does not detect image resolutions.

Code: Select all

$regexp = '#(data-src|src|src2|href)="([^"]+\.('.implode('|',$ext).')(\?[^"]+)?)"#';
In words, any asset starting with data-src=, src=, src2= or href= and with extensions .css, .js, .woff, .woff2, .ttf, .jpg, .webp, .png or .mp3
(you can add or remove any if you wish in that regex)

That will produce records like:

Code: Select all

    [0] => Array
        (
            [0] => href="catalog/view/javascript/bootstrap/css/bootstrap.min.t1535802328.css"
            [1] => href
            [2] => catalog/view/javascript/bootstrap/css/bootstrap.min.t1535802328.css
            [3] => css
        )

Code: Select all

    [6] => Array
        (
            [0] => src="catalog/view/javascript/jquery/jquery-2.1.1.min.js"
            [1] => src
            [2] => catalog/view/javascript/jquery/jquery-2.1.1.min.js
            [3] => js
        )

Code: Select all

    [66] => Array
        (
            [0] => src="image/ci/ca/pi/65/659/659_L1-Prehnite-Pendant-in-18-Carat-White-Gold-120x90.webp"
            [1] => src
            [2] => image/ci/ca/pi/65/659/659_L1-Prehnite-Pendant-in-18-Carat-White-Gold-120x90.webp
            [3] => webp
        )
etc.

There is no detection on resolution.
Ones detected, you can filter those assets on the content of the filename if it contains the resolution in case of images.

The only assets the code cannot detect are those not referenced in your html output but are referenced inside asset files.
For example, fontawesome references font files inside its css files, the code detects the css file but not the font files inside, those the code cannot see and those you should define statically.

I see your output uses srcset=, perhaps add that to the regex and see if it takes those assets as well.
or try a regex for images like:

Code: Select all

[^"'=\s]+\.(jpe?g|png|gif)

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by AndreyPopov » Sat Jul 18, 2020 4:21 am

letxobnav wrote:
Fri Jul 17, 2020 11:39 pm

The only assets the code cannot detect are those not referenced in your html output but are referenced inside asset files.
For example, fontawesome references font files inside its css files, the code detects the css file but not the font files inside, those the code cannot see and those you should define statically.
made. but fonts name always same. it's easy.

letxobnav wrote:
Fri Jul 17, 2020 11:39 pm
There is no detection on resolution.
I know.
letxobnav wrote:
Fri Jul 17, 2020 11:39 pm
Ones detected, you can filter those assets on the content of the filename if it contains the resolution in case of images.
problem in that algorithm not detect (find) mobile view images.

for example:
- in desktop view product image file name : product-250x250.jpg
algorithm find it and push

- in mobile view product image file name : product-500x500.jpg
but ……. algorithm find product-250x250.jpg, push product-250x250.jpg
not find product-500x500.jpg!
browser say: product-250x250.jpg preloaded but NOT USE in few seconds
and browser load product-500x500.jpg
double load on mobile view: first product-250x250.jpg, second product-500x500.jpg :(

may be product-500x500.jpg hide inside code and algorithm cannot find it?

temporally solve :

Code: Select all

if ( \Journal3::getInstance()->document->isMobile() ) $file = str_replace('250x250','500x500',$file);

New member

Posts

Joined
Sat Feb 04, 2017 2:53 am

Post by letxobnav » Sat Jul 18, 2020 7:54 am

may be product-500x500.jpg hide inside code and algorithm cannot find it?
as I said:
I see your output uses srcset=, perhaps add that to the regex and see if it takes those assets as well.

you can do the images separately by using:

Code: Select all

			$imageExtensions	= array('jpg','webp','svg','png','gif');
			preg_match_all('/[^"=\s]+\.('.implode('|',$imageExtensions).')/', $this->output, $match_images, PREG_SET_ORDER);
			error_log(print_r($match_images,true));

that should also cover your srcset images and give you records like:

Code: Select all

    [141] => Array
        (
            [0] => https://www.priazha-shop.com/image/cache/catalog/main/priazha-main1a-1812x468.png
            [1] => png
        )

    [142] => Array
        (
            [0] => https://www.priazha-shop.com/image/cache/catalog/main/priazha-main1a-1812x468.png
            [1] => png
        )

    [143] => Array
        (
            [0] => https://www.priazha-shop.com/image/cache/catalog/main/priazha-main1a-retina-3626x937.png
            [1] => png
        )

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by AndreyPopov » Sat Jul 18, 2020 5:21 pm

letxobnav wrote:
Sat Jul 18, 2020 7:54 am

you can do the images separately by using:

Code: Select all

			$imageExtensions	= array('jpg','webp','svg','png','gif');
			preg_match_all('/[^"=\s]+\.('.implode('|',$imageExtensions).')/', $this->output, $match_images, PREG_SET_ORDER);
			error_log(print_r($match_images,true));
yes, this code find all possible images, but …….
for desktop view output only 250x250 images
for mobile view output only 500x500 images

then I must sort and filter array for desktop view and mobile view. no good solution :( more memory and more time

New member

Posts

Joined
Sat Feb 04, 2017 2:53 am

Post by AndreyPopov » Wed Feb 24, 2021 10:38 pm

is a way to detect what dpi screen used and preload:
- only 1x image form normal dpi
- only 2x image for high dpi screen
?

New member

Posts

Joined
Sat Feb 04, 2017 2:53 am
Who is online

Users browsing this forum: No registered users and 55 guests