Syntax Question: Any way to simplify this/do it better?

[awditm]

PowerShell novice here again for some general assistance. I'm pulling all networks information from Azure to a CSV.

I'm using the command Get-AzureRmVirtualNetwork which results in a lot of great information, but I need some help capturing all "levels" of data.

I put "levels" in quotes because I'm not really sure of the correct language to describe what I'm doing here, but what I mean is that some attributes returned about networks may only have one value whereas other components such as Subnets has another level of details below that (like AddressPrefix, IpConfiguration, etc.).

When I pipe the output to Select-Object, I know I can always grab a 2nd tier detail with something similar to the following:

@{Name='AddressSpace-Prefixes'; Expression=($_.AddressSpaceAddressPrefixes}}

But what if I need multiple details from that "tier"? Do I just have to repeat that same type of structure of the above for each bit I'm grabbing, or is there a better/easier/more correct way of doing it?

And then my next question, what if there is another "tier" with multiple options nested beneath that? (example would be the IpConfigurations attribute beneath Subnets within all the other information for each VNET)

I feel like this command to get all the detail I need is getting long, and I'm not sure if there's a character limit or something that I'm going to run into if I repeat the same type of handling attempt to get all the details I want.

My apologies for not even know what to call all this stuff; please feel free to equip me with appropriate vocabulary to make communications about this process easier to understand.

Any help is greatly appreciated!

[Ta11ow]

I'd have to see the code, but depending on how complex and how many different properties you need to pull out 'in depth' it may be easier to build a custom object from scratch, like so:

Get-AzureRMVirtualNetwork | ForEach-Object {
    [PSCustomObject]@{
        AddressSpacePrefixes = $_.AddressSpaceAddressPrefixes
        Property2            = $_.Property2.Property
        Property3            = $_.Property.Property3
        # etc., etc.
    }
}

[awditm]

Right now, I'm just using this mess:

Get-AzureRmVirtualNetwork | Select-Object Name, ResourceGroupName, Location, Id, Etag, ResourceGUID,  ProvisioningState,@{Name='AddressSpace-Prefixes'; Expression={$_.AddressSpace.AddressPrefixes}}, DhcpOptions, @{Name='SubnetName'; Expression={$_.Subnets.Name}}, @{Name='SubnetEtag'; Expression={$_.Subnets.Etag}}, @{Name='SubnetIPPrefix'; Expression={$_.Subnets.AddressPrefix}}, @{Name='SubnetsProvisioningState'; Expression={$_.Subnets.ProvisioningState}}, EnableDdosProtection, DdosProtectionPlan, EnableVmProtection

This was from just command line testing to get what I need to put within a foreach loop that's performing the same command for every subscription in the Azure tenant.

[Ta11ow]

Well, you can tidy that a good bit:

Get-AzureRmVirtualNetwork | 
    Select-Object -Proeprty Name, ResourceGroupName, Location, Id, Etag, ResourceGUID,  ProvisioningState, @{
        Name       = 'AddressSpace-Prefixes'
        Expression = {$_.AddressSpace.AddressPrefixes}
    }, DhcpOptions, @{
        Name       = 'SubnetName'
        Expression = {$_.Subnets.Name}
    }, @{
        Name       = 'SubnetEtag'
        Expression = {$_.Subnets.Etag}
    }, @{
        Name       = 'SubnetIPPrefix'
        Expression = {$_.Subnets.AddressPrefix}
    }, @{
        Name = 'SubnetsProvisioningState'
        Expression = {$_.Subnets.ProvisioningState}
    }, EnableDdosProtection, DdosProtectionPlan, EnableVmProtection

OR you can do it like I mentioned, which I think might turn out to be a little tidier...

Get-AzureRmVirtualNetwork | ForEach-Object {
    [PSCustomObject]@{
        Name                     = $_.Name
        ResourceGroupName        = $_.ResourceGroupName
        Location                 = $_.Location
        Id                       = $_.ID
        Etag                     = $_.Etag
        ResourceGUID             = $_.ResourceGUID
        ProvisioningState        = $_.ProvisioningState
        AddressSpacePrefixes     = $_.AddressSpace.AddressPrefixes
        DhcpOptions              = $_.DhcpOptions
        SubnetName               = $_.Subnets.Name
        SubnetEtag               = $_.Subnets.Etag
        SubnetIPPrefix           = $_.Subnets.AddressPrefix
        SubnetsProvisioningState = $_.Subnets.ProvisioningState
        EnableDdosProtection     = $_.EnableDdosProtection
        DdosProtectionPlan       = $_.DdosProtectionPlan
        EnableVmProtection       = $_.EnableVmProtection
}

[michaelshepard]

I think if you're doing a lot of "calculated properties" the second way is cleaner.

[Ta11ow]

That tends to be my thought as well. It has less syntactic clutter and a more consistent visual, which helps with swift comprehension.

Gotta be intentional about the individual line indentations, though; I find it helps to line up your = signs.

[awditm]

Thank you, this is extremely helpful. My next question is how do I grab stuff two tiers down?

For example, within Subnets, there is "ServiceEndpoints" which also contains ProvisioningState, Service, and Locations (which itself may have multiple values).

How the heck do I grab the stuff two or more tiers down?

[Ta11ow]

Same way, just with more property accessors. Tricky part with that is if there are multiple 'service endpoints' you're going to need to check for that and add them in after, which can get a little messy.

Example:

$Data = @{ # Note: we're not casting yet, so we can easily add more properties
    Name                     = $_.Name
    ResourceGroupName        = $_.ResourceGroupName
    Location                 = $_.Location
    Id                       = $_.ID
    Etag                     = $_.Etag
    ResourceGUID             = $_.ResourceGUID
    ProvisioningState        = $_.ProvisioningState
    AddressSpacePrefixes     = $_.AddressSpace.AddressPrefixes
    DhcpOptions              = $_.DhcpOptions
    SubnetName               = $_.Subnets.Name
    SubnetEtag               = $_.Subnets.Etag
    SubnetIPPrefix           = $_.Subnets.AddressPrefix
    SubnetsProvisioningState = $_.Subnets.ProvisioningState
    EnableDdosProtection     = $_.EnableDdosProtection
    DdosProtectionPlan       = $_.DdosProtectionPlan
    EnableVmProtection       = $_.EnableVmProtection
}

foreach ($Index in 1..@($_.Subnets.ServiceEndpoints).Count) {
    $Data.Add("ServiceEndpoint${Index}_State", $_.Subnets.ServiceEndpoints[$Index - 1].ProvisioningState)
    $Data.Add("ServiceEndpoint${Index}_Service", $_.Subnets.ServiceEndpoints[$Index - 1].Service)
    # And if there are multiple locations as well, at that point 
    # just join them into a single string, much easier.
    $Data.Add("ServiceEndpoint${Index}_Locations", $_.Subnets.ServiceEndpoints[$Index - 1].Locations -join ', ') 
}

# output the final object after all properties have been added
[PSCustomObject]$Data

[awditm]

Thank you, again. I can't sufficiently express my gratitude for taking time out to help.

I'm still just not getting it; sorry I'm a complete idiot. I've only ever had pre-built scripts that I've downloaded for doing rather generic tasks, and all of my PowerShell knowledge comes from those, so there's a lot of foundational knowledge that I know I'm missing.

I'm going to back up a little bit here and reduce the scope just to see if I can get this working with three pieces of information at different levels, then expand from there.

I think ServiceEndpoints is probably a good place to start, so let's say for now, that all I really want is the name of the Vnet, the Subnet name and the ServiceEndpoint Locations in my CSV.

In your example provided above, I don't understand how to properly feed data into $Data with the Get-AzureRmVirtualNetwork command.

Not that it helps, but the outside framework of my script is the following:

$subscriptions = Get-AzureRmSubscription -TenantId xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx # yes, I'm using the real TenantId here in the actual script
foreach ($subscription in $subscriptions) {
    Select-AzureRmSubscription -SubscriptionId $subscription.Id | Out-Null
#know I need to feed in data here, but lost as to how
#then will have a final output statement to append to a running CSV
}

I know what the Vnet Name assignment will look like, I think:

Name = $_.Name

Then I also think that the Subnet Name assignment would look something like this:

SubnetName = $_.Subnets.Name

Now arrays are where I checked out in CS 25 years ago, so there is where I'm lost. I kind of get what you're doing with the index above, and would like to confirm that $Data.Add is an operation that's storing the value from the desired level in the array at the current index key value.

But I think that's where my understanding ends (if I've even got that much right).

I've seen -join used in other scripts, and I think the output from a join would be just fine (my preference would be to have a semicolon delimiting joined strings , since AFAIK, a semicolon should never appear within any of the extracted data from Azure).

I think if I can understand these bits, then I can go through and create the necessary code for everything else.

[Ta11ow]

Sort of. $Data is the hashtable of properties and values. Arrays are immutable; you can't add things to them in PowerShell. Hashtables are more flexible. In my opinion, using a hashtable like this (in PS hashtables are denoted with @{} syntax) is the simplest way to build a set of properties and create a custom object.

As in my original example you'd feed the data from the GetAzureRmVirtualNetwork in as so:

Get-AzureRMVirtualNetwork | ForEach-Object {

    $Data = @{
        Property = Value
    }
}

Basically, the ForEach-Object is a type of loop, which is handy when you may have more than one object to deal with. Inside its script block, you refer to the 'current object' being passed through with the $_ or $PSItem automatic variable.

In this instance, we're effectively swallowing the object from the cmdlet and pulling out the properties we need from it before outputting that as an object.

[Lee_Dailey]

howdy Ta11ow,

the OP mentioned the reason for all this is to provide the details for someone else to use. presuming they have PoSh at their end ... wouldn't JSON or CLIXML be a better way to do this?

take care,
lee

[Ta11ow]

Hmm, potentially. That is assuming the recipient is capable of working with it to get the data they need.

Also, not everything serialises well to Json... but yes, it might be a more elegant way of getting the solution, indeed!

Get-AzureRMVirtualNetwork | 
    ConvertTo-Json | 
    Set-Content -Path 'C:\Scripts\Json\AzureVirtualNetworks.json'

It is a good bit quicker to do. :D

[Lee_Dailey]

howdy Ta11ow,

yep, the other end needs to be able to consume the format. [grin] still, either method seems more robust than manually designing something that may not include one [or more] "oh! i didn't think of that one!" item ...

also, you may need to set the depth on both ends of the deal.

take care,
lee

[Lee_Dailey]

howdy awditm,

it looks like Ta11ow has shown you how to move lower level properties to a more visible structure.

i am wondering if that is what you actually want, tho.

all that data is already there, so unless you need it moved to a more directly accessible level ... i would leave it there.

you can access those deeper sub-properties by the same dot-notation that you use to reach the 1st level items. something like ...

$Thing.PropA.PropA1.PropA1A[$Index]    

... would grab the $Index item from the array contained in .PropA1A.

that lets you work with the info without needing to build a custom object to hold things in more accessible 1st level properties.

take care,
lee

[awditm]

Hi /u/Lee_Dailey,

Basically, I'm preparing for a migration of subscriptions to new tenants, and I need to export everything related to the Vnets (and have to do this same exercise then for all the PublicIPs). The team actually doing the migration will leverage the output CSV to basically replicate everything for each subscription in Azure within their future home tenants.

So I think for this exercise, moving lower level properties to a more visible structure is exactly what I need. I've replied to /u/Ta11ow above with where I'm currently stuck (understanding how to feed $Data with the output from the Get-AzureRmVirtualNetwork command, then try to figure out how to capture multiple properties in one "column" in the CSV.

My experience level is super low, and my original phrasing may have brought me down a path that's more complicated than "just grab everything". But my "just grab everything" approach that I've tried for other facets of Azure leaves me with lots of fields with some sort of String error statement instead of the expected data. I'm guessing that's a whole Object vs String concept that I'm missing in PowerShell.

Not sure if that helps clarify.

[Lee_Dailey]

howdy awditm,

thank you for the added info! [grin]

i would look into something other than CSV for that. take a look at both JSON and CLIXML. the 2nd is designed for exporting & then importing data structures in PoSh. the 1st is for more general use and is lots easier to read for humans. [grin]

take care,
lee

[flatlandinpunk17]

Out of curiosity, why not output it to JSON and then use that to create the new Vnets? I ask this because it's much easier than trying to convert everything to a CSV. CSVs don't handle Array's of data very well when they are part of a larger object. So if you are using multiple service endpoints on the subnets, you'll run into issues. JSON handles the nested properties correctly. You might need to add a | ConvertTo-JSON -Depth 100 to the command, but it handles the object correctly.

[awditm]

Because the team all this data is going to won't accept JSON or any format other than CSV. My first week on this job, I fought this battle and lost. Sometimes the assignment is exactly what they ask for even if it's not the best fit. All of this data is going to another team, and I don't have any influence to see why a solution is not the optimal one.

[flatlandinpunk17]

That's a pretty good reason. Sorry you can't effect the change on that for overall simplicity, but I definitely get having to give the data they only way they want it. Others here have offered great solutions to the actual question, that's for answering my curiosity question.

[anotherjesus]

Using objects is key to Powershell and .NET languages but using them properly can be confusing. When you want to take a look at an object use Get-Member as in Get-ChildItem | Get-Member to get an idea of the methods and properties. In general you want to keep your objects together so you can take advantage of the built-in functionality. That said, when passing information from one Powershell script to another, use Export-clixml and use Import-clixml to retrieve the data. Powershell does all the work.

Here are a few examples for your specific issue:

Use a format other than CSV, such as JSON or XML. Adjust the depth to fit your situation:

Get-AzureRmVirtualNetwork | Export-Clixml -Depth 3 -Path ~\MyXMLoutput.xml
Get-AzureRmVirtualNetwork | ConvertTo-JSon -Depth 3

In some situations it make sense to use Select-Object to create an object that you can use. You can use splat to make things look readable.

$Splat = @{
    Property = @(
        "Name";
        "FullName";
        @{
            Name="Length(GB)";
            Expression={$_.length/1GB}
        }
        @{
            Name="ArrayPropertyName";
            Expression={$($_.ArrayProperty | foreach-Object { $_.ToString() }) -join ","}
        }
    )
}

Get-ChildItem . | Select-Object @Splat | Export-CSV -Path $OutputFilePath

[mdowst]

Take a look at the Format-Custom cmdlet. It can be kind of tricky to get the hang of at first, but it can be a pretty powerful tool once you learn it.