In addition to wget, curl is a primary tool in the linux guru’s tool chest when webscraping, poking around the internet or intranet, downloading files, testing applications and making web calls via a CLI tool.
Curl is actively developed by a community of developers and has its web presence at https://curl.haxx.se/.
A Quick look at some Feature Highlights of Curl:
- config file support
- multiple URLs in a single command line
- range “globbing” support: [0-13], {one,two,three}
- multiple file upload on a single command line
- custom maximum transfer rate
- FTP download, authentication
- Kerberos 5 (*14)
- active/passive using PORT, EPRT, PASV or EPSV
- single file size information (compare to HTTP HEAD)
- upload via http-proxy as HTTP PUT
- download / upload resume
- all operations can be tunneled through a http-proxy
- customizable to retrieve file modification date
- SCP both password and public key auth
- SFTP both password and public key auth
- SFTP with custom commands sent before/after the transfer
- TELNET connection negotiation
- custom telnet options
- TELNET stdin/stdout I/O
- full LDAP URL support
- SMBv1 over TCP and SSL
- SMB download & upload
- SMB authentication with NTLMv1
- SMTP authentication: Plain, Login, CRAM-MD5, Digest-MD5, NTLM (*9), Kerberos 5, (*4) and External.
- send e-mails
- mail auth support for trusted server-to-server relaying
- SMTP multiple recipients
- POP3 authentication: Clear Text, APOP and SASL
- SASL based authentication: Plain, Login, CRAM-MD5, Digest-MD5, NTLM (*9), Kerberos 5 (*4) and External.
- list e-mails
- retrieve e-mails
For more features
Latest Curl Release
The latest version of curl as of Jan 2020 is 7.68.0. Downloads are posted on the curl site and the source code can be found on the github page for the project. See the Changelog for the latest curl Release Notes.
How to build and install Curl from source
First get the source code for curl from the download page. I won’t demo this because likely you need curl or wget to get the source code and that is redundant to show in this tutorial. See the downloaded software below:
Also you will need to install GCC and make on your machine before starting, let’s assume its Debian or compatible:
$ apt-get install gcc $ apt-get install make
Now unpack the software with the tar command. And then enter the unpacked directory and run configure and make and make install
$ tar xzvf curl-7.68.0.tar.gz $ cd curl-7.68.0 $ ./configure $ make $ sudo make install
If you get this warning, you may have 2 versions of curl installed and need to ensure you are using the libcurl you just compiled with the curl version you compiled. So set the LD_LIBRARY_PATH to fix it:
WARNING: curl and libcurl versions do not match. Functionality may be affected. $ export LD_LIBRARY_PATH=/usr/local/lib
Now you can see curl and libcurl have matching versions and curl is working as expected.
How to install Curl on Debian 10 or Ubuntu 19.10
Curl is installed by default on Debian and Ubuntu latest versions. If you have removed it for some reason you can install it with apt-get
sudo apt-get install curl
How to install Curl on CentOS 8
Curl is installed by default on CentoOS and Red Hat latest versions. If you have removed it for some reason you can install it with yum
sudo yum install curl
Download WebPage with Curl Example
Let’s download a simple webpage with curl. Here is the syntax and example:
curl https://linuxhint.com/sitemap.xml > out.webpage head out.webpage
Multi-Page Download Example
You can use wildcards and fancy syntax to download many files at once. Here is an example to download from Linuxhint the sitemap pages which follow a regular pattern. The -O option will store the files in names that are the same as the original web pages but locally.
curl -O https://linuxhint.com/sitemap-pt-post-2019-[01-12].xml
Conclusion
Curl is mostly installed on most modern Linux distributions by default so getting access to it is super easy. Compilation or package install is also easy. Check the resources below for more details on all the abundant features of curl.
More Curl Resources
- Curl Manpage
- Free Curl e-Book Everything Curl
- Use Curl in Bash Scripts for advanced usage
