Search This Blog

Showing posts with label PowerShell. Show all posts
Showing posts with label PowerShell. Show all posts

2026-09-25

PowerShell - Getting Remote MS SQL Server 2017 SSL Certificate Using Low Level Network Communication

Product: Microsoft MS SQL Server
Version: 2017

Overview

There are few scenarios where you would like to download the SSL certificate and save as file while working with MS SQL Server client or server programs.  I come into situation where the old version of IBM Cognos Analytics required to add remote MS SQL Server's self-signed SSL cert into its own Java keystore.  The challenge is that the MS SQL Server does not has any self-signed SSL cert, and it dynamically creates a SSL cert (MS called it as fallback SSL cert) in memory during MS SQL Server startup.  So it is not possible to save the certificate file.

Following are few more scenarios where you might want to save the remote MS SQL Server's SSL cert:

  1. MS SQL Server DBA does not has Windows admin privilege to run Windows Certificate Manager (certmgr.msc) to save the SSL cert
  2. Windows admin is not familiar with Windows Certificate Manager (certmgr.msc) and not able to provide the SSL cert
  3. MS SQL Server DBA does not know anything about SSL fallback cert, when MS SQL Server does not has any SSL cert

The MS SQL Server API Microsoft.Data.SqlClient in any programming, e.g. Visual C#, Visual C++, Java, Python, Perl, is not possible to obtain the SSL cert.

openssl.exe utility has drop MS SQL Server TDS (Tabular Data Stream) support, so it is not possible to use this utility to get the SSL cert.

As the result, I develop a PowerShell program that uses low level TCP network communication that can download SSL certificate from local or remote MS SQL Server through TCP/IP protocol.

Technical Design

Low level TCP programming uses PowerShell System.Net.Sockets.TcpClient.  MS SQL Server login is always SSL encrypted.  It contains 2 phase, which is called PRELOGIN and LOGIN7.  During PRELOGIN phase, MS SQL Server will provide SSL certificate to the DB client program, which then can proceeds with LOGIN7 phase that will login to MS SQL Server with username and password (assumes DB authentication, not Windows user).

The most challenging part is research MS SQL Server PRELOGIN protocol, as this is poorly documented by Microsoft.  You can find the official document from learn.microsoft.com below:

Title: Open Specifications > Protocols > SQL Server Protocols > Technical Documents > [MS-SSTDS]: Tabular Data Stream Protocol Version 4.2 > Messages > Message Syntax > Packet Header Message Type Stream Definition > PRELOGIN

  • Ref: https://learn.microsoft.com/en-us/openspecs/sql_server_protocols/ms-sstds/75e62f67-f057-4d46-82b3-6920fe0ebada

The PowerShell program will make use of hex and binary to convert the PRELOGIN data structure back and forte with remote MS SQL Server server.

While reading MS' document, we need to ignore any document that tell encryption is optional, as well as TLS v1.2 is optional.  In fact, many versions of MS SQL Server have long enforced TLS v1.2, and encryption during login.  So there are many obsolete content which confuses reader like us.  The login sequence is called as Tabular Data Stream with short name of TDS.  My program is written for MS SQL Server 2017, which is using TDS v7.4, but newer MS SQL Server uses TDS v8.0.  So this program might not work with newer MS SQL Server due to a change in PRELOGIN phase.

My program set the encryption as 0x01, which in MS' doc called "Encryption is available and on."  Again, all MS SQL Server support SSL cert, so you can safely accept encryption is available.

Some variables in my program call "stream" which represent TCP network communication.  At the beginning of the low level TCP network connection, a variable of NetworkStream is created against TcpClient connection.  This is to facilitate back and fort communication with remote MS SQL Server.

Soon after open TCP connection, my program will send a static PRELOGIN to remote MS SQL Server to tell it that my program needs to login.  This part of the communication is not encrypted, and in clear text. Don't simply change this static PRELOGIN array value, as 2 of the bytes (2 array elements) contains number of bytes inside this array.  You can refer to MS doc for exact array element, if you want to change this array.

After that, MS SQL Server will reply back to tell my program its MS SQL Server version, whether it supports encryption (it will be always supported), number of bytes it will send back.  My program will retrieve this reply (MS doc called message or message stream) using 2 separate steps (command NetworkStream.Read()).  The reason is because the 1st part is a constant 8 bytes, which I called "payload header" while MS doc implies it is payload as well, with short name of PL.  My program then find out number of bytes remote MS SQL Server going to send to me, and in the 2nd part, my program will read the remaining reply (message stream).  I called the 2nd part of it as "prelogin payload option token" and "prelogin payload data."  MS doc is not clear about how they called them.  For my program, I mainly determine the remote MS SQL Server will tell my program that it accept my PRELOGIN request, and it does support SSL encryption.  This is very important as SSL cert will only be send to my program if it replied that it supports SSL encryption.  This will double confirm new MS SQL Server standard that LOGIN7 is always SSL encrypted.

The next part is the most challenging part as I need to continue to talk to MS SQL Server using TDS header structure, while enforcing SSL TLS v1.2.  For this part, I can't hard code the static array like earlier as I need to call System.Net.Security.SslStream.AuthenticateAsClient which will indirectly calls System.IO.Stream over TCP.  So there is an extra class called TdsTlsStream that helps me to perform read() and write() over the TCP network communication.  Inside this class, in read() function, it will read first 8 bytes to determine network packet length, and in write() function, it will create extra 8-bytes packets to insert before sending over the data to remote MS SQL Server.

System.Net.Security.SslStream.AuthenticateAsClient is the main call that will obtain SSL certificate from remote MS SQL Server.  Anything that is wrong in the program will leads to timeout, then remote MS SQL Server will close the network connection.  If it success, then the program will be able to extract the SSL certificate using System.Security.Cryptography.X509Certificates.X509Certificate2.

You can further modify the program to add following line to save the SSL certificate as a file locally:

E.g. Export-Certificate -Cert $<yr variable name> -FilePath db_certificate_DER_binary.cer -Type CERT

Above command will create a new SSL certificate file called db_certificate_DER_binary.cer in current directory

PowerShell Program

Following is the PowerShell program

[CmdletBinding()]
param()


# Define your SQL Server instance details, default port 1433
$SqlServer = "your-ms-sql-server-hostname-or-IP"
$SqlPort   = 20277


# Enforce TLS 1.2 for the handshake protocol selection
$TlsVersion = [System.Security.Authentication.SslProtocols]::Tls12
# --- SQL Server Pre-Login Handshake Protocol Payload ---
# This standard 8-byte TDS header tells SQL Server a client login attempt is starting.
[byte[]]$TdsPreLoginPacket = 0x12, 0x01, 0x00, 0x2F, 0x00, 0x00, 0x01,0x00, 0x00, 0x00, 0x1A, 0x00, 0x06, 0x01,0x00, 0x20,0x00, 0x01, 0x02, 0x00, 0x21, 0x00, 0x01,0x03, 0x00, 0x22, 0x00, 0x04, 0x04, 0x00,0x26, 0x00,0x01, 0xFF, 0x09, 0x00, 0x00, 0x00, 0x00,0x00, 0x01, 0x00, 0xB8, 0x0D, 0x00, 0x00,0x01
 
# Create a TCP Client to connect to the SQL port
$TcpClient = New-Object System.Net.Sockets.TcpClient
try {
    $TcpClient.Connect($SqlServer, $SqlPort)
$TcpClient.SendTimeout    = 10000;
$TcpClient.ReceiveTimeout = 10000;

    $NetworkStream = $TcpClient.GetStream()

    $NetworkStream.Write($TdsPreLoginPacket, 0, $TdsPreLoginPacket.Length)


# 3. Read the server's TDS Pre-Login Response Header
$responseHeader = New-Object byte[] 8
$bytesRead = $NetworkStream.Read($responseHeader, 0, $responseHeader.length)
if ($bytesRead -lt $responseHeader.length) {
Write-Error "PRELOGIN - $bytesRead bytes read, but expects 8 bytes"
throw "Failed to read a valid PRELOGIN TDS header response"
}

$respPayloadLength = ($responseHeader[2] -shl 8) + $responseHeader[3] - $responseHeader.length

Write-Host "--- TDS HEADER RECEIVED ---" -ForegroundColor Cyan
Write-Host "Raw Bytes Header: " ($responseHeader -join ' ')
Write-Host "Remaining Payload Length: $respPayloadLength bytes" -ForegroundColor Cyan

    # Continue to read the token data and data
if ($respPayloadLength -gt 0) {
$responsePayload = New-Object byte[] $respPayloadLength
$bytesRead = $NetworkStream.Read($responsePayload, 0, $respPayloadLength)
$intEndofTokenHeader = [array]::IndexOf($responsePayload, [byte]0xff)
$intEndofTokenData = $intEndofTokenHeader + 1
Write-Host "Finished reading payload: $respPayloadLength bytes" -ForegroundColor Cyan
Write-Host "Offset of FF terminator: " $intEndofTokenHeader -ForegroundColor Cyan
Write-Host "Raw Bytes: " ($responsePayload -join ' ') -ForegroundColor Gray
Write-Host "Raw Bytes Token Header: " ($responsePayload[0..$intEndofTokenHeader] -join ' ') -ForegroundColor Gray
Write-Host "Raw Bytes Token Data: " ($responsePayload[$intEndofTokenData..($respPayloadLength - 1)] -join ' ') -ForegroundColor Gray
}

$isServerEncrypted = 0
$posPayload = 0
while ($posPayload -lt $intEndofTokenHeader) {
$plOptionType = $responsePayload[$posPayload]
Write-Host "PRELOGIN respond header - Position $posPayload=$plOptionType"
# Exit loop if reach header terminator character 0xFF
if ($plOptionType -eq 0xFF) {
Write-Host "PRELOGIN respond header - Reached end of header"
break
}
elseif ($plOptionType -ne 1) {
$posPayload += 5
Write-Host "PRELOGIN respond header - Skipping 5 bytes to offset $posPayload"
continue
}

        # Calculate the address of encryption byte
$valueOffset = [System.BitConverter]::ToUInt16([byte[]]@($responsePayload[$offset + 2], $responsePayload[$offset + 1]), 0)
Write-Host "PRELOGIN respond header - Found encryption byte in offset address $posPayload with value $valueOffset"
$isServerEncryptedRaw = $responsePayload[$valueOffset]
$isServerEncrypted = $isServerEncryptedRaw -band 0x0F
Write-Host ("PRELOGIN respond header - offset $valueOffset's encryption byte value = {0} (0x{0:X2}) -> {0} (0x{1:X2})" -f $isServerEncrypted,$isServerEncrypted ) -ForegroundColor Cyan
break
}

Write-Verbose "Entryption is supported, so it is possible to get SSL cert. $isServerEncrypted is not 0 means DB server will perform encryption"
Write-Verbose "Setup TLS stream"
$tlsStreamToWrap = [TdsTlsStream]::new($NetworkStream)

    # Wrap the network stream in an SslStream object for the TLS handshake
# We pass a callback to automatically trust self-signed SQL dev certificates
Write-Verbose "Setup SSL stream inside TLS stream"
    $SslStream = New-Object System.Net.Security.SslStream($tlsStreamToWrap, $false, {
        param($sender, $certificate, $chain, $sslPolicyErrors)
        # Always return true to capture the certificate even if untrusted/self-signed fallback
        return $true
    })

    Write-Host "SslProtocol negotiated #1: $($sslStream.SslProtocol)"

    # Authenticate as client to trigger the handshake
Write-Verbose "Specified SSL to use $TlsVersion, and obtain remote DB server's SSL cert"
    $SslStream.AuthenticateAsClient($SqlServer, $null, $TlsVersion, $false)
    Write-Host "SslProtocol negotiated #2: $($sslStream.SslProtocol)"

    # Extract the server certificate
$ServerCertificate = [System.Security.Cryptography.X509Certificates.X509Certificate2]($SslStream.RemoteCertificate)

    if ($ServerCertificate) {
        Write-Host "`n=== SQL Server TLS 1.2 Certificate Details ===" -ForegroundColor Green
        $ServerCertificate | Format-List Subject, Issuer, SerialNumber, Thumbprint, NotBefore, NotAfter
        # Verify if it is the fallback certificate
        if ($ServerCertificate.Subject -like "*SSL_Self_Signed_Fallback*") {
$Host.PrivateData.VerboseForegroundColor = 'Cyan'
            Write-Verbose "This is the internal SQL Server Self-Signed Fallback Certificate, so it is in RAM, and will change after DB server start/restart/reboot"
        }
    } else {
        Write-Warning "Successfully connected with TLSv1.2, but no remote certificate was returned by the server."
    }


} catch {
$lineText   = $_.InvocationInfo.Line
    $lineNumber = $_.InvocationInfo.ScriptLineNumber
    $position   = $_.InvocationInfo.PositionMessage

    Write-Host "❌ Error occurred on line $lineNumber" -ForegroundColor Red
    Write-Host "Code block: $lineText" -ForegroundColor Yellow
    Write-Host "Position details:`n$position"
} finally {
    # Properly close and dispose of connections
    if ($SslStream) { $SslStream.Dispose() }
    if ($TcpClient) { $TcpClient.Close(); $TcpClient.Dispose() }
}



class TdsTlsStream : System.IO.Stream {
[System.IO.Stream]$InnerStream
[int]$PayloadLength = 0

TdsTlsStream([System.IO.Stream]$InnerStream) {
$this.InnerStream = $InnerStream
}

[bool] get_CanRead() { return $this.InnerStream.CanRead }
[bool] get_CanWrite() { return $this.InnerStream.CanWrite }
[bool] get_CanSeek() { return $this.InnerStream.CanSeek }
[Int64] get_Length() { return $this.InnerStream.Length }
[Int64] get_Position() { return $this.InnerStream.Position }
[void] set_Position([Int64]$Value) { $this.InnerStream.Position = $Value }
[int] get_ReadTimeout() { return $this.InnerStream.ReadTimeout }
[int] get_WriteTimeout() { return $this.InnerStream.WriteTimeout }

[void] Flush() { $this.InnerStream.Flush() }
[Int64] Seek([Int64]$Offset, [System.IO.SeekOrigin]$Origin) { return $this.InnerStream.Seek($Offset, $Origin) }
[void] SetLength([Int64]$Value) { $this.InnerStream.SetLength($Value) }

[int] Read([byte[]]$Buffer, [int]$Offset, [int]$Count) {
# Removes TDS header before setting the network Buffer
if ($this.PayloadLength -eq 0) {
$tlsHeader = [byte[]]::new(8)
$read = 0
while ($read -lt 8) {
$read += $this.InnerStream.Read($tlsHeader, 0, 8)
}

$lengthBeforeHeader = [System.BitConverter]::ToUInt16([byte[]]@($tlsHeader[3], $tlsHeader[2]), 0)
$lengthBeforeHeader -= 8
$this.PayloadLength = $lengthBeforeHeader
}

if ($Count -gt $this.PayloadLength) {
$Count = $this.PayloadLength
}
$read = $this.InnerStream.Read($Buffer, $Offset, $Count)
$this.PayloadLength -= $read
return $read
}

[void] Write([byte[]]$Buffer, [int]$Offset, [int]$Count) {
$newPayload = $this.GenerateTdsHeader($Buffer, $Offset, $Count)
$this.InnerStream.Write($newPayload, 0, $newPayload.Length)
}

[byte[]] GenerateTdsHeader([byte[]]$Payload, [int]$Offset, [int]$Count) {
# The length is big endian encoded so it is inserted in reverse order
$lengthBytes = [System.BitConverter]::GetBytes([uint16]($Count + 8))

$newPayload = [byte[]]::new(8 + $Count)
$newPayload[0] = 0x12  # Type - Pre-Login
$newPayload[1] = 0x01  # Status - End of message (EOM)
$newPayload[2] = $lengthBytes[1]
$newPayload[3] = $lengthBytes[0]
$newPayload[4] = 0  # SPID
$newPayload[5] = 0  # SPID
$newPayload[6] = 0  # PacketID
$newPayload[7] = 0  # Window
[System.Array]::Copy($Payload, $Offset, $newPayload, 8, $Count)

return $newPayload
}
}

2026-04-01

SAP Data Services Web Service (SOAP) Calling - Example

Product: SAP Data Services
Version: 4.2.x - 4.3.x

Overview

SAP BODS (Data Services) has DSMC browser GUI which offers web service functionality.  This allows external scheduler to easily integrate with it.

This blog post is going to show using PowerShell Invoke-WebRequest, so that you can see its  SOAP communication in low level.  For experience person, it should be easy to figure out how to configure the external job scheduler to use BODS WSDL 2.1 web service (SOAP) call.

For this example, I'm using following dummy value to illustrate

  • BODS URL: http://localhost:8080/DataServices/servlet/webservices?ver=2.1
  • Server hostname: localhost
  • Authentication Method: Build in Enterprise
  • Username: administrator
  • Password: hidden_hash_value
  • Job Server local repository name: LOCAL_REPO_DEV1
  • Job ID: 2
  • Going to simulate the OOTB web service call Get_BatchJob_Status, which will return 2 values of whether the web service call success or not (0=success), as well as job status in string of Running/Succussed/Error/Warning
    • BODS doc: https://help.sap.com/docs/SAP_DATA_SERVICES/ab33122a997f40d89e340549ff0bced8/5748962b6d6d1014b3fc9283b0e91070.html
Login is required to obtain session ID.  The session ID going to be used in subsequent web service call

For full SAP BODS web service reference doc, see https://help.sap.com/docs/SAP_DATA_SERVICES/ab33122a997f40d89e340549ff0bced8/574996556d6d1014b3fc9283b0e91070.html

Using PowerShell Invoke-WebRequest to Call BODS Web Service

1. Login

$headers1 = @{"SOAPAction" = "function=Logon"}

$soapBody1 = @"

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://www.businessobjects.com/DataServices/ServerX.xsd">

   <soapenv:Header/>

   <soapenv:Body>

      <ser:LogonRequest>

         <username>administrator</username>

         <password>hidden_hash_value</password>

         <cms_system>localhost</cms_system>

         <cms_authentication>secEnterprise</cms_authentication>

      </ser:LogonRequest>

   </soapenv:Body>

</soapenv:Envelope>

"@


$response1 = Invoke-WebRequest -UseBasicParsing -Method Post -ContentType "text/xml;charset=UTF-8" -Headers $headers1 -Body $soapBody1 http://localhost:8080/DataServices/servlet/webservices?ver=2.1

[xml]$xmlResponse1 = $response.Content

$dsmc_SessionID = $xmlResponse1.Envelope.Body.session.SessionID

$dsmc_SessionID


2. Call Get_BatchJob_Status

$headers2 = @{"SOAPAction" = "jobAdmin=Get_BatchJob_Status"}

$soapBody2 = @"

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://www.businessobjects.com/DataServices/ServerX.xsd">

   <soapenv:Header>

      <ser:session>

         <SessionID>$dsmc_SessionID</SessionID>

      </ser:session>

   </soapenv:Header>

   <soapenv:Body>

      <ser:batchJobStatusRequest>

         <runID>2</runID>

         <repoName>LOCAL_REPO_DEV1</repoName>

      </ser:batchJobStatusRequest>

   </soapenv:Body>

</soapenv:Envelope>

"@


$response2 = Invoke-WebRequest -UseBasicParsing -Method Post -ContentType "text/xml;charset=UTF-8" -Headers $headers2 -Body $soapBody2 http://localhost:8080/DataServices/servlet/webservices?ver=2.1

[xml]$xmlResponse2 = $response2.Content

$xmlResponse2.Envelope.Body.batchJobStatusResponse


localtypes                                              returnCode status
----------                                              ---------- ------
http://www.businessobjects.com/DataServices/ServerX.xsd 0          error

Using PowerShell New-WebServiceProxy to Call BODS Web Service

This is using SOAP web service call.

$url_dsmc = "http://localhost:8080/DataServices/servlet/webservices?ver=2.1&wsdlxml"
# First call to BODS web service server to get the wsdl
$response1 = New-WebServiceProxy -Uri $url_dsmc -Namespace dsmc_namespace

$dsmc_namespace = $dsmc.GetType().Namespace
$dsmc_conn_ops = New-Object -TypeName "$dsmc_namespace`.Connection_Operations"

$param = New-Object "dsmc_namespace.LogonRequest"
$param.username = "administrator"
$param.password = "hidden_hash_value"
$param.cms_system = "localhost"
$param.cms_authentication = "secEnterprise"

# 2nd call - Triggers logon, and it will return Session ID
# Stores the session_id for next call
$session_id = $dsmc_conn_ops.Logon($param)

# Next is to check Job status
# Local repo: LOCAL_REPO_DEV1
# Job's run ID 2
$job_request = New-Object "dsmc_namespace.batchJobStatusRequest"
$job_request.repoName = "LOCAL_REPO_DEV1"
$job_request.runID = 2
$dsmc_Batch_Job_Admin = New-Object -TypeName "$dsmc_namespace`.Batch_Job_Admin"
$dsmc_Batch_Job_Admin_namespace = $dsmc_conn_ops.GetType().Namespace
$dsmc_Batch_Job_Admin.sessionValue = $session_id

# 3rd time calling BODS SOAP web service
$dsmc_Batch_Job_Admin.Get_BatchJob_Status($job_request)

Output
returnCode status
---------- ------
         0 error

Analysis

The parameter and web service methods required to referring to BODS web service wsdl.  For example:
  1. New-Object "dsmc_namespace.LogonRequest" - wsdl contains "<xsd:element name="LogonRequest">" with complexType, and 4 string parameters of username , password , cms_system , cms_authentication 
  2. Logon() - wsdl indicated that the pass-in parameter is call "LogonRequest" directly under the root. It does not belongs to any class (PowerShell), or wsdl:binding (wsdl)
  3. PowerShell needs to call New-Object -TypeName $dsmc_namespace`.<wsdl:binding> to switch between 5 different binding of Batch_Job_Admin, Repo_Operations, Batch_Jobs, Connection_Operations, Realtime_Service_Admin (default)

2022-02-03

Powershell: List AD Group for Specific User and Domain

Product: Windows
Utility: Powershell, ActiveDirectory

Windwos AD domain users who are not administrator of ActiveDirectory users are often getting confusing push back from AD admin when the business users are requesting for access that controlled by AD.

This article is going to show various commands that business users (end users) who are not AD admin, and allows them to find out which AD group they belongs to directly, or indirectly (Windows term if group nesting).

CMD

Following command will display all the nesting group that a user belongs to

1. net user <username> /domain

Limitation #1: You cannot change the domain, if the computer you login to belongs to multiple domain

Limitation #2: Doesn't show child groups. It will always display direct groups the user belongs to

Limitation #3: Display "Description" name of the group, not the actual group name

Limitation #4: Group name is truncated after 21 characters

2. gpresult /v

Limitation #1: can't specify which domain if you belongs to multiple AD domain

Limitation #2: Cannot show direct group. It will always display nesting child groups

3. whoami /groups

Limitation #1: can't specify which domain if you belongs to multiple AD domain

Limitation #2: Cannot show direct group. It will always display nesting child groups

PowerShell

Note: Windows 10 - 11 required to use Settings > Optional features to add module "RSAT: Active Directory Domain Services and Lightweight Directory Services Tools"

1. Show AD group the current login user directly belongs to:

(Get-ADuser $env:username -server $env:userdomain -Properties memberof).memberof | get-adgroup -server $env:userdomain | select-object -ExpandProperty name

Limitation #1: It cannot show nesting group

1.1. Show AD group of username tester1

(Get-ADuser  tester1 -server $env:userdomain -Properties memberof).memberof |  get-adgroup -server $env:userdomain | select-object -ExpandProperty name

1.2. Similar to above, but sorted

(Get-ADuser  tester1 -server $env:userdomain -Properties memberof).memberof |  get-adgroup -server $env:userdomain | select-object -ExpandProperty name | sort

2. Show AD group user in specific domain, e.g. domain GAME-ADM

(Get-ADuser $env:username -server GAME-ADM -Properties memberof).memberof | get-adgroup -server GAME-ADM | select-object name

3. Show AD group user GAMER1 in domain GAME-ADM

(Get-ADuser GAMER1 -server GAME-ADM -Properties memberof).memberof | get-adgroup -server GAME-ADM | select-object -ExpandProperty name

4. Show AD group is managed by who

General syntax: Get-ADGroup [group name] -server [domain name] -properties ManagedBy

Example: group name GAMEVIPUSR, domain name GAME-ADM, the command will be

Get-ADGroup GAMEVIPUSR -server GAME-ADM -properties ManagedBy

Get-ADGroup GAMEVIPUSR -server $env:userdomain -properties ManagedBy

5. Show members belong to AD group SAP-ADMIN

Get-ADGroupMember SAP-Admins  -server $env:userdomain | select-object -ExpandProperty name

6. Similar to #5, but don't display user. Only display child AD group that belongs to it

Get-ADGroupMember SAP-Admins  -server $env:userdomain | where-object objectClass -ne "user" | select-object -ExpandProperty name

7. Show a child group TESTER-GRP belongs to which parent group

Get-ADPrincipalGroupMembership GWRE-CASE-DataManagement -server $env:userdomain | select-object -ExpandProperty name

8. Show a child group TESTER-GRP belongs to which parent group like "PROD*" and not case-sensitive

Get-ADPrincipalGroupMembership GWRE-CASE-DataManagement -server $env:userdomain | | where name -match "PROD*" | select-object -ExpandProperty name 


2020-11-24

PowerShell: Download Video in Parallel

Product: PowerShell

PowerShell script which shown an example to download video from web, which contains multi-part TS video.

Assumptions:

  • The TS video starts with number 1
  • Last part of the video is number 600
  • The file name is a running number that increment by 1
  • All videos have exact same URL, but different by running number
  • The filename has 3 character digit, e.g. 001, 002, 003
  • The output file is call S01E08-[part].ts
  • The video extension is .ts
  • 5 parallel download
Start-Job -Scriptblock {
	1..100 | foreach {
        $url_main="https://www.szhysws.com:65/20190422/RuBaJpqw/2000kb/hls/1svhk1647"
		$i="{0:000}" -f $_
		$url="$url_main$i.ts"
		$file="S01E08-$i.ts"
		Write-Output $file
		Invoke-WebRequest -UserAgent "Mozilla/5.0 (Windows NT 6.1; Win64; x64)" -Uri $url -OutFile $file
	}
}

Start-Job -Scriptblock {
	101..200 | foreach {
        $url_main="https://www.szhysws.com:65/20190422/RuBaJpqw/2000kb/hls/1svhk1647"
		$i="{0:000}" -f $_
		$url="$url_main$i.ts"
		$file="S01E08-$i.ts"
		Write-Output $file
		Invoke-WebRequest -UserAgent "Mozilla/5.0 (Windows NT 6.1; Win64; x64)" -Uri $url -OutFile $file
	}
}

Start-Job -Scriptblock {
	201..300 | foreach {
        $url_main="https://www.szhysws.com:65/20190422/RuBaJpqw/2000kb/hls/1svhk1647"
		$i="{0:000}" -f $_
		$url="$url_main$i.ts"
		$file="S01E08-$i.ts"
		Write-Output $file
		Invoke-WebRequest -UserAgent "Mozilla/5.0 (Windows NT 6.1; Win64; x64)" -Uri $url -OutFile $file
	}
}

Start-Job -Scriptblock {
	301..400 | foreach {
        $url_main="https://www.szhysws.com:65/20190422/RuBaJpqw/2000kb/hls/1svhk1647"
		$i="{0:000}" -f $_
		$url="$url_main$i.ts"
		$file="S01E08-$i.ts"
		Write-Output $file
		Invoke-WebRequest -UserAgent "Mozilla/5.0 (Windows NT 6.1; Win64; x64)" -Uri $url -OutFile $file
	}
}

Start-Job -Scriptblock {
	401..600 | foreach {
        $url_main="https://www.szhysws.com:65/20190422/RuBaJpqw/2000kb/hls/1svhk1647"
		$i="{0:000}" -f $_
		$url="$url_main$i.ts"
		$file="S01E08-$i.ts"
		Write-Output $file
		Invoke-WebRequest -UserAgent "Mozilla/5.0 (Windows NT 6.1; Win64; x64)" -Uri $url -OutFile $file
	}
}

2020-11-22

PowerShell: Download TS Video Series from URL with Running Number

Product: Microsoft PowerShell
Version: 7.0

If you need to download a video of type TS extension, and it has the same URL, but each file has a running sequential number, this PowerShell script will download everyone into local machine, and you can specify the output file name as desire

Get_Video1.ps1
$url_main="https://www.szhysws.com:65/20190422/BS7i3zoM/2000kb/hls/83Fe3552"
1..600 | foreach {
$i="{0:000}" -f $_
 $url="$url_main$i.ts"
 $file="S01E01-$i.ts"
 Write-Output $file
 Invoke-WebRequest -UserAgent "Mozilla/5.0 (Windows NT 6.1; Win64; x64)" -MaximumRetryCount 100 -Uri $url -OutFile $file }

Remove "-MaximumRetryCount 100" for PowerShell version 3-4, or if it doesn't support this parameter

2020-04-23

PowerShell: Unhide Files

Product: PowerShell
Version: 1.0 above

Following command can be used to unhide files and directory, excluding

  • thumbs.db
  • desktop.ini

Get-ChildItem -Force -Recurse | Where-Object{ $_.Attributes.ToString().Contains(“System”) -or $_.Attributes.ToString().Contains(“Hidden”) -and ($_.Name -ne “thumbs.db”)  -and ($_.Name -ne “desktop.ini”) } | foreach {$_.Attributes = “Archive”}

or

Get-ChildItem -Force -Recurse | Where-Object{ $_.Attributes.ToString().Contains(“System”) -or ( $_.Attributes.ToString().Split(“, “) -contains “Hidden”) -and ($_.Name -ne “thumbs.db”)  -and ($_.Name -ne “desktop.ini”) } | foreach {$_.Attributes = “Archive”}



Following command can be used to unhide all files and directory

Get-ChildItem -Force -Recurse | Where-Object{ $_.Attributes.ToString().Contains(“System”) -or  ($_.Attributes.ToString().Contains(“Hidden”) } | foreach {$_.Attributes = “Archive”}

or

Get-ChildItem -Force -Recurse | Where-Object{ $_.Attributes.ToString().Contains(“System”) -or  ($_.Attributes.ToString().Split(“, “) -contains “Hidden”) } | foreach {$_.Attributes = “Archive”}

2018-07-20

PowerShell: Download Movies from IPTV http Stream Using Invoke-WebRequest

Product: PowerShell
Version: 2

It is often that IPTV provider which provides http URL to access the m3u file will contain following output:

#EXTINF:-1,RAAZI (2018)
http://server.internext.tv:8080/movie/4621036/4621036/29542.mp4
#EXTINF:-1,Pari (2018)
http://server.internext.tv:8080/movie/4621036/4621036/28274.mkv
...cut...

This post will show how to use PowerShell to download them in series.  The reason not download in parallel is because IPTV provider often limit number of concurrent connections

1. Convert the m3u file to CSV file, so that it looks like this

RAAZI (2018),http://server.internext.tv:8080/movie/4621036/4621036/29542.mp4
Pari (2018),http://server.internext.tv:8080/movie/4621036/4621036/28274.mkv

2. Adds a header called OutFile and Uri.  This will be how it looks like

OutFile,Uri
RAAZI (2018),http://server.internext.tv:8080/movie/4103600/4103699/29542.mp4
Pari (2018),http://server.internext.tv:8080/movie/4103600/4103699/28274.mkv

3. Adds the full directory to save the file, as well as extension, e.g. D:\Movies.  Keep the extension identical to the input video stream

OutFile,Uri
D:\Movies\RAAZI (2018).mp4,http://server.internext.tv:8080/movie/4103600/4103699/29542.mp4
D:\Movies\Pari (2018).mkv,http://server.internext.tv:8080/movie/4103600/4103699/28274.mkv

4. Save the file, e.g. download1.txt
5. Download the movies in sequence

import-csv download1.txt | foreach-object {  write-host $_.OutFile;  Invoke-WebRequest -UserAgent "Mozilla/5.0 (Windows NT 6.1; Win64; x64)" -Uri $_.Uri -OutFile $_.OutFile -verbose }

2016-12-15

PowerShell: Rename File but Keep Extension Name

Rename all the files in current directory by removing the whitespace character from the base filename
Dir | where { ! $_.PSIsContainer } | Rename-Item –NewName { $_.Basename.TrimEnd() + $_.extension } -verbose

Same as above, but includes all the subdirectories
Dir -recurse | where { ! $_.PSIsContainer } | Rename-Item –NewName { $_.Basename.TrimEnd() + $_.extension } -verbose

Rename all the files in current directory by appending "_old" to the end of the base filename
Dir -recurse | where { ! $_.PSIsContainer } | Rename-Item –NewName { $_.name + "_old" + $_.extension } -verbose

Rename all the files including subdirectories by inserting "old_" to the beginning of the base filename
Dir -recurse | where { ! $_.PSIsContainer } | Rename-Item –NewName { "old_" + $_.name + $_.extension } -verbose

Rename all the files including subdirectories by replacing "new" with "old" in the base filename
Dir -recurse | where { ! $_.PSIsContainer } | Rename-Item –NewName { $_.basename.replace("new","old") + $_.extension } -verbose

Rename all the *.trp files including subdirectories by removing the "00" at end of filename, i.e. replacing "00.trp" with ".trp" in the base filename
Dir -recurse -filter *.trp | Rename-Item –NewName { $_.name –replace “00.trp“,”.trp” } -verbose

Rename all the MP4 files by first 8 characters of the base filename
Dir *.mp4 | Rename-Item -NewName { $_.basename.substring(0,7) + $_.extension } -verbose
Dir *.mp4 | Rename-Item -NewName { "流光引.EP" + $_.basename.substring(27,2) + $_.extension } -verbose -WhatIf

Rename all the files in current directory with substring starting location 9, plus 15 characters of the base filename
Dir -recurse | where { ! $_.PSIsContainer } | Rename-Item –NewName { $_.name.SubString(8,15) + $_.extension } -verbose

Reaname all the files with extension webm in current directory with substring starting 8 and 38, and takes 4 characters.  Keeping the same extension
Dir ENG*.webm | Rename-Item -NewName { $_.basename.substring(8,4) + $_.basename.substring(38,4) + $_.extension } -verbose -WhatIf
 

 

2016-11-13

PowerShell: Rename File Name To Directory Name

Product: PowerShell

This command will generate a script which can be used to rename multiple files in the directory to the directory name.  This is used when:

1. Inconsistent filename in the directory, each with unique extension
2. Directory name is unique, but filename is random, so you would like to make the filename to be same as directory name

I have a directory which contains following file extensions used for the purpose of this post:

Dir: 4Ever
Files: file1.trp, file1.txt, file1.idx, file1.nfo

get-childitem  Stockings | rename-item  -NewName { $_.directory.name + $_.extension }

Output
PS I:\SVHD_VIDEO> dir 4Ever


    Directory: I:\SVHD_VIDEO\4Ever


Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---        11/13/2016   9:20 AM 1020313600 4Ever.trp
-a---        11/13/2016   9:20 AM     857792 4Ever.idx
-a---        11/13/2016   9:20 AM       4228 4Ever.ifo
-a---        11/13/2016   9:20 AM       1480 4Ever.txt

For renaming across multiple directories, use following syntax

get-childitem  -recurse | where { ! $_.PSIsContainer } | rename-item  -NewName { $_.directory.name + $_.extension }

where PSIsContainer means it is a directory.  Exclamation mark (!) used to exclude directory

2013-03-19

PowerShell: Equivalent to UNIX find/grep

Product: Windows PowerShell

I was trying to find out the PowerShell command that equivalent to following UNIX command to search through all files for a particular text (or regular expression)

find . | grep -i "family_picture"

The PowerShell equivalent is

Get-ChildItem -recurse | select-string -pattern "family_picture"

Default is not case sensitive, so no need to map "grep -i" parameter. If needed case sensitive, then it will be

Get-ChildItem -recurse | select-string -pattern "family_picture" -casesensitive

2013-03-01

PowerShell: Rename Directory To File Name

Product: PowerShell

This command will generate a script which can be used to rename a directory to txt file residing inside the parent directory. This used the full path name

I have a directory which contains 1 txt file used to label the purpose of the directory.

G:\Recovered\recup_dir.1\fat32\007785\Argon_000.txt

G:\Recovered\recup_dir.1\fat32\0217CE\Superman_000.txt

get-childitem -recurse -filter *.txt | %{Write-Host Rename-Item $_.directory $_.basename }

Rename-Item G:\Recovered\recup_dir.1\fat32\007785 Argon_000
Rename-Item G:\Recovered\recup_dir.1\fat32\0217CE Superman_000
Rename-Item G:\Recovered\recup_dir.1\fat32\02B84B Anony_000
Rename-Item G:\Recovered\recup_dir.1\fat32\031CD8 IronMan_000
Rename-Item G:\Recovered\recup_dir.1\fat32\038977 Goats_000

This is a similar command, but will use relative name of the directory


get-childitem -recurse -filter *.txt | %{Write-Host Rename-Item $_.directory.basename $_.basename }

Rename-Item 007785 Argon_000
Rename-Item 0217CE Superman_000
Rename-Item 02B84B Anony_000
Rename-Item 031CD8 IronMan_000
Rename-Item 038977 Goats_000
It is nice to save this into a batch file for audit as well as historical troubleshooting purpose

Following command is similar to 2nd command, but it will execute immediately instead of displaying only

get-childitem -recurse -filter *.txt | %{ rename-item -path $_.directory -NewName $_.basename -verbose }

While finding out the correct syntax to dynamically rename the directory, I used follwoing


Get-ChildItem -recurse -include *.txt | select-object -property directory, basename


Directory                                                   BaseName
---------                                                   --------
G:\Recovered\recup_dir.1\fat32\[007785]                     Argon_000
G:\Recovered\recup_dir.1\fat32\[00CB99]                     Superman_000
G:\Recovered\recup_dir.1\fat32\[01A7E1]                     Anony_000
G:\Recovered\recup_dir.1\fat32\[0217CE]                     IronMan_000
G:\Recovered\recup_dir.1\fat32\[02B84B]                     Goats_000

Other commands used

PS G:\Recovered\recup_dir.1> Get-ChildItem -recurse -include *.txt | get-member
PS G:\Recovered\recup_dir.1\fat32> Get-ChildItem -recurse -include *.txt | select-object directory, name

2013-02-28

PowerShell: Rename directory with square bracket

Product: PowerShell

I have a corrupted partition, and after recovered with GetDataBack, all the directories becomes a hexcode enclosed with square bracket.

So I used following command to remove the square bracket, which is used as a wild-card in PowerShell. Surprisingly this trick is hard to find in Internet


ls -filter *`[*`]* | move-item -dest {$_ -replace '\[|\]',''}

Explanation:
1. ls command filter directory or file name containing square bracket
2. Inside move-item -dest, I used escape character \ followed by left and right square bracket
3. I used or operator | to replace both square bracket characters

2013-01-26

PowerShell: Rename File With Regular Expression

ls *.rmvb | %{$_.name -replace 'TVBOXNOW ','' -replace '_1280x720',''}

From:
TVBOXNOW 大太監 CH18_1280x720.rmvb
TVBOXNOW 大太監 CH19_1280x720.rmvb
TVBOXNOW 大太監 CH20_1280x720.rmvb
TVBOXNOW 大太監 CH21_1280x720.rmvb

To:
大太監 CH18.rmvb
大太監 CH19.rmvb
大太監 CH20.rmvb
大太監 CH21.rmvb
大太監 CH22.rmvb
大太監 CH23.rmvb
大太監 CH24.rmvb
大太監 CH25.rmvb


Rename by removing following:
1. "TVBOXNOW "
2. "_1280x720"

ls *.rmvb | %{ren $_  $($_.name -replace 'TVBOXNOW ','' -replace '_1280x720','')}

2011-10-31

Genesys Log Analysis Tips #2: PowerShell

Software Component: Genesys Outbound Contact Server

Following example demonstrate searching OCS log for list of campaign. This can easily be changed to display CPD port, agent, extension, calling list, etc.

This command search log gad_1_20111031_000000_001_1.log with following criteria:
1. Contain word "Name:GSW_CAMPAIGN_NAME" which basically capture all outbound campaign name
2. Select the output, and hide the line number. Select-String will always return file name, line number, and the search result, which is too long. In this example, I only searching 1 file, so file name is not important
3. Sort the output alphabetically
4. Display unique campaign name, so that duplicate campaign name won't be shown


Select-String -path gad_1_20111031_000000_001_1.log -pattern "Name:GSW_CAMPAIGN_NAME" | select-object Line | Sort-Object Line -Unique

This can provide a quick glance on what campaign is running, and interact by agents in GAD


Line
----
.Name:GSW_CAMPAIGN_NAME Value:Inbound
.Name:GSW_CAMPAIGN_NAME Value:Test_Campaign_1
.Name:GSW_CAMPAIGN_NAME Value:Test_Campaign_2
.Name:GSW_CAMPAIGN_NAME Value:Test_Campaign_3
.Name:GSW_CAMPAIGN_NAME Value:Test_Campaign_4
.Name:GSW_CAMPAIGN_NAME Value:Test_Campaign_5
.Name:GSW_CAMPAIGN_NAME Value:Test_Campaign_6
.Name:GSW_CAMPAIGN_NAME Value:Test_Campaign_7
.Name:GSW_CAMPAIGN_NAME Value:Test_Campaign_8
.Name:GSW_CAMPAIGN_NAME Value:Test_Campaign_9
.Name:GSW_CAMPAIGN_NAME Value:Test_Campaign_a
.Name:GSW_CAMPAIGN_NAME Value:Test_Campaign_b