{"id":8134,"date":"2021-08-13T01:08:04","date_gmt":"2021-08-13T06:08:04","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=8134"},"modified":"2021-08-13T01:08:04","modified_gmt":"2021-08-13T06:08:04","slug":"python-selenium-headed-v-s-headless","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=8134","title":{"rendered":"Python Selenium Headed v\/s Headless"},"content":{"rendered":"<p>We are automating a file download &#8212; it works fine when running headed, but headless execution doesn&#8217;t manage to log in. Proxying the requests through Fiddler show that several JavaScript pages download unexpected content.<\/p>\n<p><a href=\"https:\/\/www.rushworth.us\/lisa\/?attachment_id=8135\" rel=\"attachment wp-att-8135\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-large wp-image-8135\" src=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2021\/08\/requestRejected-1024x73.png\" alt=\"\" width=\"960\" height=\"68\" srcset=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2021\/08\/requestRejected-1024x73.png 1024w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2021\/08\/requestRejected-300x21.png 300w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2021\/08\/requestRejected-768x55.png 768w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2021\/08\/requestRejected-750x53.png 750w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2021\/08\/requestRejected.png 1155w\" sizes=\"auto, (max-width: 960px) 100vw, 960px\" \/><\/a><\/p>\n<p>I&#8217;ve added a user-agent to the request, but I&#8217;ve noticed that the ChromeDriver also sets sec-ch-* headers &#8230; I expect the null sec-ch-ua causes the web server to refuse our request. I don&#8217;t see any issues in the ChromeDriver repo for the sec-ch-* headers &#8230; and I don&#8217;t really want to walk back versions until I find one that doesn&#8217;t try setting this header value. Firefox&#8217;s GeckoDriver, though, doesn&#8217;t set them &#8230; so I moved the script over to use Firefox instead of Chrome and am able to download the file.<\/p>\n<p>Headed run:<\/p>\n<p>GET \/o\/telx-theme\/css\/A.bootstrap.css+slick,,_slick.css,Mcc.JKqfH-juDS.css.pagespeed.cf.ZO22sEGAvO.css HTTP\/1.1<br \/>\nHost: example.com<br \/>\nConnection: keep-alive<br \/>\nsec-ch-ua: &#8220;Chromium&#8221;;v=&#8221;92&#8243;, &#8221; Not A;Brand&#8221;;v=&#8221;99&#8243;, &#8220;Google Chrome&#8221;;v=&#8221;92&#8243;<br \/>\nsec-ch-ua-mobile: ?0<br \/>\nUser-Agent: &#8220;Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/85.0.4183.102 Safari\/537.36<br \/>\nAccept: text\/css,*\/*;q=0.1<br \/>\nSec-Fetch-Site: same-origin<br \/>\nSec-Fetch-Mode: no-cors<br \/>\nSec-Fetch-Dest: style<br \/>\nReferer: https:\/\/example.com\/web\/guest\/login<br \/>\nAccept-Encoding: gzip, deflate, br<br \/>\nAccept-Language: en-US,en;q=0.9<br \/>\nCookie: JSESSIONID=0330C2C988F31010790779A126EA6F55.node1; COOKIE_SUPPORT=true; GUEST_LANGUAGE_ID=en_US; AWSELB=039B496118DDEAD697B2B51C93852940763289C324F9E7C7223F953330AF5506573D13C4D5599541FD3CADB645303C1CAEB6D26992826965DA6C8BEDBDE9C297AE26CD76ED; AWSELBCORS=039B496118DDEAD697B2B51C93852940763289C324F9E7C7223F953330AF5506573D13C4D5599541FD3CADB645303C1CAEB6D26992826965DA6C8BEDBDE9C297AE26CD76ED; TS0194d418=01092b79076749232d762d2a6c232e015d103453fbeda3826bd3d20e1d937f5a90cabe03655c97a79198969eea539e4c2e7fc426216092c78ccda85763d52300ce05672704e45b4fc25516d2c24279656db7b0242f7c8b9c8bfed35b7608afb0c54bbc33d489f431059d048094c1e707a20d28031885ca6c61f81613ac299044f0c2b9ba36<\/p>\n<p>&nbsp;<\/p>\n<p>Headless run:<\/p>\n<p>GET \/o\/telx-theme\/css\/A.bootstrap.css+slick,,_slick.css,Mcc.JKqfH-juDS.css.pagespeed.cf.ZO22sEGAvO.css HTTP\/1.1<br \/>\nHost: example.com<br \/>\nConnection: keep-alive<br \/>\nsec-ch-ua:<br \/>\nsec-ch-ua-mobile: ?0<br \/>\nUser-Agent: Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/92.0.4515.131 Safari\/537.36<br \/>\nAccept: text\/css,*\/*;q=0.1<br \/>\nSec-Fetch-Site: same-origin<br \/>\nSec-Fetch-Mode: no-cors<br \/>\nSec-Fetch-Dest: style<br \/>\nReferer: https:\/\/example.com\/web\/guest\/login<br \/>\nAccept-Encoding: gzip, deflate, br<br \/>\nAccept-Language: en-US<br \/>\nCookie: JSESSIONID=F4293ECE33B134CC368C0E62D6923B48.node1; COOKIE_SUPPORT=true; GUEST_LANGUAGE_ID=en_US; AWSELB=039B496118DDEAD697B2B51C93852940763289C324A5AB24AE470C70960B5319A93C181302D27B4C9425A4AA05795334C4404D491FBCC8E6A9B809746A802EAC2EC8C2FBFA; AWSELBCORS=039B496118DDEAD697B2B51C93852940763289C324A5AB24AE470C70960B5319A93C181302D27B4C9425A4AA05795334C4404D491FBCC8E6A9B809746A802EAC2EC8C2FBFA; TS0194d418=01ba3b12a4ef612e3839114024b5082fd19d56b17293c914ff867740ad37ae362e385934695ad3fc275074bfd1ee24c7d1591b146ad39d153a8758aecc8eb44d374dc1c689e540deca9566f723df65e9f5ad26551e25bacd5df14e4e6104a91a0ecdb59a65176bd5a0ebed284847e0e6618a05ed1d9db6b544e195d8e1f41164e7199a6596<\/p>\n","protected":false},"excerpt":{"rendered":"<p>We are automating a file download &#8212; it works fine when running headed, but headless execution doesn&#8217;t manage to log in. Proxying the requests through Fiddler show that several JavaScript pages download unexpected content. I&#8217;ve added a user-agent to the request, but I&#8217;ve noticed that the ChromeDriver also sets sec-ch-* headers &#8230; I expect the &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[33],"tags":[45,664,1442],"class_list":["post-8134","post","type-post","status-publish","format-standard","hentry","category-coding","tag-coding","tag-python","tag-selenium"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/8134","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=8134"}],"version-history":[{"count":1,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/8134\/revisions"}],"predecessor-version":[{"id":8136,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/8134\/revisions\/8136"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=8134"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=8134"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=8134"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}