Pfthb

In ruby, I'm doing an HTTP GET request to a website that uses SSL, and for some reason it only gives me the data I need when proxying it through an HTTP proxy (burp suite).

For example:

 if uri.port == 443
 response = Net::HTTP.start(uri.host, uri.port, "172.16.38.182", "8080", :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) 
 else
 response = Net::HTTP.start(uri.host, uri.port) 
 end

by proxying this through BurpSuite, I can see the data that I need. But if I try this below:

 if uri.port == 443
 response = Net::HTTP.start(uri.host, uri.port, :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) 
 else
 response = Net::HTTP.start(uri.host, uri.port) 
 end

then I get hex characters that looks like this:

Any idea why this would be the case? Is it because of the specific SSL/TLS version used or something and the web application proxy knows how to interact with it but net/https doesn't? can't figure this out.

Turns out the reason this looks the way it does is because it's returned with gzip encoding.

I was able to successfully decode it by using the following:

 if response['content-encoding'] == 'gzip'
 gz = Zlib::GzipReader.new(StringIO.new(response.body.to_s))
 uncompressed_string = gz.read
 end