How to increase server response time ?
Ping from Fremont to Atlanta: 60ms
curl1 from Fremont to Atlanta: 137ms
Initial curl (https) from Fremont to Atlanta: 700ms
Subsequent curl (https) from Fremont to Atlanta: 468ms
Ping from Dallas to Atlanta: 22ms
curl (http) from Dallas to Atlanta: 50ms
Initial curl (https) from Dallas to Atlanta: 322ms
Subsequent curl (https) from Dallas to Atlanta: 229ms
Testing from different locations around the country will yield different results, and latencies from outside of the country will definitely be higher. Different SSL ciphers and certificates, user-agents, and server configurations can also wildly affect performance. Latency issues bother me too—especially that 700ms initial connection time from Fremont—but at the end of the day there's only so much tuning you can do. The best solution is to just have servers "near" your users, wherever they may be, which is why worldwide CDNs are popular!
Here's the same test from within Atlanta:
Ping from Atlanta to Atlanta: 0.45ms
curl (http) from Atlanta to Atlanta: 6ms
Initial curl (https) from Atlanta to Atlanta: 136ms
Subsequent curl (https) from Atlanta to Atlanta: 13ms
You can see that ~131ms "TCP ping" (i.e. the HTTP request) delta (Fremont-Atlanta cf. Atlanta-Atlanta) is translating into ~564ms of initial TLS negotiation delta. This is largely due to the number of roundtrips (four, including the HTTP request/response, I think) it takes between client and server to fully negotiate TCP and TLS handshakes. Even with things like OCSP stapling, TLS false start, and TCP smart connect (which are supposed to reduce the number of roundtrips—testing with openssl s_time instead of curl gives more promising results), it still quickly adds up to a ton of time. This is why HTTP keep-alive is so important: once the client has "paid" the "cost" of TLS negotiation on the first request, it can just keep the connection open and make subsequent requests much more quickly.