Showing posts with label Scripting. Show all posts
Showing posts with label Scripting. Show all posts

Tuesday, 19 May 2020

youtube-dl (how to) - best guide to get you started

####################################################################
###        youtube-dl - the best guide to get you started        ###
###                                                              ###

>> Installing
[Note]: Before you start; 
If you have upgraded to Catalina (on never have installed xcode and gcc before), then you will need to:
install Xcode (from Apple's App Store), then run the command: xcode-select --install 
then install gcc, and then install youtube-dl

[TIP]: You must run brew update first. Then you can either use the brew upgrade to update all packages, or, run brew upgrade youtube-dl to update only youtube-dl

//To install youtube-dl [1] under MacOS use:
brew install youtube-dl

//Once it is installed, update the application:
sudo youtube-dl -U

Sunday, 5 April 2020

Ping an IP range from the command line interface (CLI)

If you ever wanted to do host discovery while no specific network scanning tools were available (e.g. nmap), or you wanted to avoid creating a script file (e.g. due to having no write permissions)? Here are a couple of useful commands depending on the OS of your choice:

Windows OS:
for /L %i IN (1,1,254) DO ping -n 1 -w 1 192.168.1.%i| FIND /i "Reply" >> IPs.txt

Note: command prompt has a limitation when asked to return the exact string using regex and returns the whole line. Here is an example of replacing find with findstr in order to use a regular expression (regex). 
findstr /r "[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*"

The regular expression can still be useful in cases such as:
ipconfig | findstr /r "[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*"


Linux OS:
for i in {1..254}; do (ping -c 1 -i 1 192.168.1.$i >/dev/null && echo "192.168.1.$i" &); done

Note: The above command will only list the discovered IP address, without any additional text.

Tip/Trick: Did you know you can use apr to achieve the same results but much faster. This seems to be reliable under Linux. The following command will list the discovered host on your network, including any additional information per IP (including the IPv6 address).
for ip in $(seq 1 254); do arp -n 192.168.1.$ip | grep on; done

The following command however, will list only the IP addresses without any additional text (also avoiding to specify the IP range twice)
for ip in $(seq 1 254); do arp -n 192.168.1.$ip | grep on | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' ; done





HOST DISCOVERY WITHOUT NMAP OR SCRIPT FILES
===========================================

Useful when no scanning tools are installed, or you can't write a script
file (e.g. no write permissions). Set the network prefix once at the start
of each command (n= / net=) instead of editing the IP in several places.

Key idea: a ping sweep finds hosts that answer ICMP. Many hosts block ICMP
(Windows Firewall does by default), but every host on your local subnet
must still answer ARP. So run the sweep first, then read the ARP cache:
that catches the hosts ping alone misses. ARP results only cover your own
local subnet; hosts beyond a router only show up via ping.


-------------------------------------------------------------------------
WINDOWS (cmd)
-------------------------------------------------------------------------

1) Ping sweep, IP addresses only, saved to IPs.txt:

for /L %i in (1,1,254) do @for /f "tokens=3 delims=: " %a in ('ping -n 1 -w 300 192.168.1.%i ^| find "TTL="') do @echo %a>>IPs.txt

Notes:
- -w is in milliseconds (-w 1 = 1 ms, far too short). 200-500 ms suits a LAN.
- Filtering on "TTL=" instead of "Reply" avoids false positives such as
  "Reply from 192.168.1.1: Destination host unreachable", and also works
  on non-English Windows.
- In a .bat file, write %%i and %%a instead of %i and %a.
- The loop is sequential, so a full /24 can take about a minute.

2) Then add hosts that blocked ping but answered ARP:

arp -a | findstr /i "dynamic"

3) findstr and regular expressions:
findstr has no "print only the match" option (no equivalent of grep -o);
it always returns the whole line. It also doesn't support {n} quantifiers.
It is still useful for matching lines, e.g.:

ipconfig | findstr /r "[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*"

To extract only the IP, use "for /f" tokenizing (as in #1) or PowerShell.


-------------------------------------------------------------------------
WINDOWS (PowerShell 5.1+, parallel, finishes in seconds)
-------------------------------------------------------------------------

1) Ping sweep, IP addresses only:

$n='192.168.1'; $t=1..254|%{(New-Object Net.NetworkInformation.Ping).SendPingAsync("$n.$_",500)}; [Threading.Tasks.Task]::WaitAll($t); $t.Result|?{$_.Status -eq 'Success'}|%{$_.Address.IPAddressToString}

2) ARP cache afterwards (IP + MAC):

Get-NetNeighbor -AddressFamily IPv4 -State Reachable,Stale,Delay,Probe | ? IPAddress -like "$n.*" | select IPAddress,LinkLayerAddress


-------------------------------------------------------------------------
LINUX (bash)
-------------------------------------------------------------------------

1) Parallel ping sweep, IP addresses only, sorted:

net=192.168.1; for i in {1..254}; do (ping -c1 -W1 $net.$i &>/dev/null && echo $net.$i) & done | sort -t. -k4n; wait

Notes:
- -W is the reply timeout in seconds. (-i is the interval between pings,
  which does nothing with -c 1.)
- "wait" stops results printing after the prompt returns, and stops
  redirected/piped output from being cut off.

2) Sweep + ARP cache (recommended; also catches ICMP-blocking hosts):

net=192.168.1; for i in {1..254}; do ping -c1 -W1 $net.$i &>/dev/null & done; wait; ip -4 neigh show to $net.0/24 | awk '!/FAILED|INCOMPLETE/{print $1}' | sort -t. -k4n

To include MAC addresses, replace {print $1} with {print $1, $5}.

Notes on ARP:
- arp / ip neigh send nothing; they only read the local cache of hosts
  your machine has recently talked to. That's why they are run after a
  sweep, not instead of one.
- The extra column is the MAC (hardware) address, not an IPv6 address.
- "arp" (net-tools) is deprecated on modern Linux; use "ip neigh".
- Dump the table once instead of calling arp 254 times.

3) Bonus: IPv6 discovery in one packet (replace eth0 with your interface):

ping -6 -c2 ff02::1%eth0 >/dev/null; ip -6 neigh show dev eth0

This pings the link-local all-nodes multicast address, then lists the
IPv6 neighbors that answered.

Portability:
- {1..254} and &> are bash-specific. Under plain sh, use $(seq 1 254)
  and >/dev/null 2>&1.


-------------------------------------------------------------------------
SCANNING MORE THAN ONE /24 (RANGES THAT CROSS OCTETS)
-------------------------------------------------------------------------

Two cases to keep separate:

  A) Whole subnets, e.g. 192.168.2.0 through 192.168.5.255
     -> just nest a loop over the third octet.

  B) A precise range, e.g. 192.168.2.3 through 192.168.5.67
     -> convert start/end to integers, loop over the integers, convert
        each back to a dotted IP. This is the only clean way to start and
        stop partway through a subnet.

A full multi-subnet sweep launches hundreds or thousands of pings. Throttle
the parallelism (xargs -P on Linux, the async batch on Windows), and expect
it to take longer and put noticeable traffic on the wire.


--- Linux (bash) ---

A) Whole subnets 192.168.2-5.x, sweep + ARP cache:

net=192.168; for o3 in {2..5}; do for o4 in {1..254}; do ping -c1 -W1 $net.$o3.$o4 &>/dev/null & done; done; wait; ip -4 neigh show | awk '!/FAILED|INCOMPLETE/{print $1}' | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n

B) Precise range 192.168.2.3 - 192.168.5.67, throttled with xargs:

s=$(( (192<<24)+(168<<16)+(2<<8)+3 )); e=$(( (192<<24)+(168<<16)+(5<<8)+67 )); for ((i=s;i<=e;i++)); do printf '%d.%d.%d.%d\n' $((i>>24&255)) $((i>>16&255)) $((i>>8&255)) $((i&255)); done | xargs -P256 -I{} sh -c 'ping -c1 -W1 {} >/dev/null 2>&1 && echo {}' | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n

Change the four numbers in s=(...) and e=(...) to set the start and end IP.
After it finishes, read the ARP cache to add ICMP-blocking hosts:

ip -4 neigh show | awk '!/FAILED|INCOMPLETE/{print $1}' | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n


--- Windows (PowerShell 5.1+) ---

Paste these two helpers into the session once:

function ip2int($ip){ $b=$ip -split '\.'; [uint32]$b[0]*16777216 + [uint32]$b[1]*65536 + [uint32]$b[2]*256 + [uint32]$b[3] }
function int2ip([uint32]$i){ '{0}.{1}.{2}.{3}' -f (($i -shr 24) -band 255),(($i -shr 16) -band 255),(($i -shr 8) -band 255),($i -band 255) }

Precise range 192.168.2.3 - 192.168.5.67, parallel:

$s=ip2int '192.168.2.3'; $e=ip2int '192.168.5.67'; $ips=for($i=$s;$i -le $e;$i++){int2ip $i}; $t=$ips|%{(New-Object Net.NetworkInformation.Ping).SendPingAsync($_,500)}; [Threading.Tasks.Task]::WaitAll($t); 0..($t.Count-1)|?{$t[$_].Result.Status -eq 'Success'}|%{$ips[$_]}

For whole subnets just pass '192.168.2.0' and '192.168.5.255' as start/end.
A for-loop counter is used instead of the .. range operator on purpose:
these IPs exceed int32, which .. can't handle in PowerShell 5.1.


--- Windows (cmd) ---

Whole subnets 192.168.2-5.x, IPs to IPs.txt (nested for /L):

for /L %j in (2,1,5) do @for /L %i in (1,1,254) do @for /f "tokens=3 delims=: " %a in ('ping -n 1 -w 300 192.168.%j.%i ^| find "TTL="') do @echo %a>>IPs.txt

In a .bat file use %%j, %%i and %%a. For a precise start/end that isn't a
whole subnet (starting at .3, ending at .67), use the PowerShell version
above - integer math is far cleaner there than in cmd.


-------------------------------------------------------------------------
macOS / BSD
-------------------------------------------------------------------------

- ping -W is in milliseconds on macOS, so use -W 1000.
- ARP cache, IP addresses only:

arp -an | grep -v incomplete | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}'

(The original "grep on" filter matches this BSD/macOS output format,
e.g. "? (192.168.1.1) at aa:bb:cc:dd:ee:ff on en0", but not Linux output.)


Saturday, 28 March 2020

Command line (CLI) check and monitor COVID-19 statistics

Use the following command to quickly check for COVID-19 (corona virus) statistics from the console (CLI), for the countries you want to monitor.


Command line:
clear && curl -s https://corona-stats.online/?minimal=true | grep "Rank\|(DE)\|(IT)\|(ES)\|(GR)\|(US)\|(AE)\|(GB)\|(CH)\|(AT)\|(NL)"


Modify the command as needed with any valid ISO 3166-1 Alpha-2 Country Codes. 

Thanks goes to @ekrysis for collecting the data sources and providing the means to query the information. See more at GitHub: https://github.com/sagarkarira/coronavirus-tracker-cli 

Some examples:
# Get data from worldometers api i.e source 2
curl https://corona-stats.online