You can get all the functions on a Composer page and then iterate through them to find the number of layout functions.

There is a blog post the EDB/DropIt blog.
http://labs.dropit.se/blogs/post/2009/11/09/How-to-determine-if-an-EPiServer-Composer-function-is-placed-in-another-Composer-function.aspx

Not exactly what you want, but the approach can be useful. Good luck!

// Make sure that the current page is an Composer page
        if (this.CurrentExtensionPageData != null)
        {
            ContentFunctionData currentContentFunctionData = new ContentFunctionData(this.ContentFunctionLink);
           
            // Loop through all content areas in the page
            foreach(ContentAreaData ca in this.CurrentExtensionPageData.ContentAreas)
            {   
                // If any of the content areas contains the current content function it means that the function is placed directly on the page
                // - it's not placed in another content function
                if (ca.ContentFunctions.Contains(currentContentFunctionData))
                    return false;
            }
        }
       
        // If we reach this code it means that the content function is placed in another content function
        return true;