Repository navigation
Add Tools with Copy site structure - #56
Open
Gabriel Mongefranco (gabrielmongefranco) wants to merge 46 commits into
Open
Gabriel Mongefranco (gabrielmongefranco) wants to merge 46 commits into
Gabriel Mongefranco (gabrielmongefranco) wants to merge 46 commits into
Conversation
When both copy methods fail, the status message now says to select the ID shown and press Ctrl+C (Cmd+C on a Mac) instead of asking the person to copy it manually. The clipboard tests and the usage page now expect and describe the new message. Nothing else about copying changes. Closes #52
Both Refresh sites list controls used to remove the saved sites and empty the list before asking SharePoint search again, so when both the sorted request and the unsorted retry failed, the list and its saved copy were gone and Choose a site said no sites had been found yet. The loader now keeps the sites from the last successful load in memory and in browser storage until a new answer arrives, replaces them only when SharePoint answers (including with an empty list), and resolves to a loaded or kept outcome. The helper text under the Site address box is now a live status region, and it and the Choose a site status line say that the list couldn't be refreshed or loaded and what to do next. The typed address, the Find a site text, the open dialog, and focus are left as they were, and a second refresh while one is running shares the request. Playwright tests cover a failed refresh at both controls, a first-ever failure, a retry that works, an empty answer, and blocked browser storage. The usage, troubleshooting, and development pages describe the behavior.
Sitenalyzer built links to site pages and shared items with encodeURI, and built top-level folder links from the raw server-relative path, so a name with a # became a fragment and a name with a literal % or %23 was read as an escape. A new helper, snaEncodeServerRelativePath, encodes each folder and file name of a path SharePoint returns as typed, such as FileRef and ServerRelativeUrl, and keeps the slashes between them. The page inventory, the shared file and folder lookup, the top-level folder links, and the agent and site skill rows all use it now. Folder links are also kept only when they are inside the site being read, as page and shared item links already were. Addresses SharePoint has already formed, such as DefaultViewUrl, are left as they are. Site pages, the page details, and the print document now show each page's address with the names as typed, and only the link uses the encoded form. This keeps a value in a file name, such as an email address, hidden like any other text while Show hidden values is off. A browser test supplies pages, folders, a shared file, and a shared folder whose names hold #, %, the text %23, spaces, an apostrophe, and an accented letter, and checks the links on screen and in the print document. The test list in docs/development.md describes it. A reviewer should open a real site with a page and a folder whose names contain # and %, and check that Open page and the folder link open the right item in a new window with SharePoint's page router turned off.
Search this report now finds a list, library, column, or term by a word that is only in its description, with the same quoted-phrase and wildcard rules as names. Each search index entry keeps the saved description, and the search hides values that look like personal information in it before comparing, as it already does for names and synonyms, so a hidden phone number cannot be found while Show hidden values is off. A description saved in the editor is searchable at once because the index is rebuilt after the save; a draft that was not saved is not indexed. The search results help and the usage page now mention descriptions, and the tests cover a keyword found only in a list, a column, and a term description, quoted phrases and wildcards in descriptions, a hidden phone number in a description, and the saved and unsaved description cases.
When a view was replaced, Sitenalyzer kept each table's filter text but not its sort order, so a sorted table came back in its usual order after opening another section or list tab, or after Show technical details redrew the view. Each table's sort order is now remembered by its table ID in memory, next to its filter text, and set again when the table is rebuilt. A table that was never sorted opens in its usual order. A remembered sort by a column the table can't show now, such as a technical column while Show technical details is off, is left out when the table is built, and comes back when the column is shown again. Refresh report, Change site, and Start over clear the remembered order with the rest of the report. No browser storage is used. The new browser tests cover the restored order, sort indicator, and filter text; another table staying in its usual order; keyboard sorting after the order is restored; a sort by a technical column; and Refresh report and Start over. Please check by hand with a screen reader that a restored heading announces its sort order, and check the change on a SharePoint page with ?csp=enforce.
The filter box above a table compared each row's address in its encoded form. Since page, folder, and shared item links now encode every part of the path, an @ in a file name became %40, so the check for hidden values no longer saw an email address there. Typing part of that address into the Site pages or Top-level folders filter box found the row while Show hidden values was off. The filter now decodes every address field before it hides values and compares, the way agent and site skill addresses were already handled. Links keep their encoded addresses. A new test types part of a hidden email address into the Site pages and Top-level folders filter boxes and checks that nothing matches until Show hidden values is on. It fails without this change. Refs #42
This adds a Tools section to the report screen, at #sna/tools, with one part, Copy site structure, at #sna/tools/copy. It is the base that the rest of issue #34 builds on: the screen, the copy plan, the order of the copy steps, the check of the destination afterwards, and the results. The screen has a source (this site's report, or a data dictionary file), a destination address with Choose a site and Check destination, a table of lists and libraries with a new name and address for each, the term sets of the site's own term group, four switches (Site pages, Site Assets, Agents and site skills, and Permissions, which is off by default), and a Replace checkbox. The destination check refuses an address on another host before sending any request, and refuses a site where the person isn't a site collection administrator. Copy to destination reads what the copy needs from this site, asks the person to confirm, runs the steps, reads the destination back, and shows a results table that Save results (.xlsx) saves. Stop copying, Start over, Change site, and Refresh report stop a copy before its next request. The code is in src/tools/. core.js has the one helper that writes to a site, plan.js reads full list settings, columns with SchemaXml, views, content types, site columns, and term records, runner.js runs the steps, verify.js checks the destination, and ui.js draws the screen. Creating lists and libraries works now. Term sets, columns, views, content types, files, pages, permissions, and reading a data dictionary file are not built yet: their files hold the functions the steps call, and each item they would copy shows as Skipped with a note. In sitenalyzer.js, request digests are now kept for each site, because SharePoint accepts a digest only on the site that issued it, and a write asks for a new digest once when SharePoint refuses one. The existing description saves and $batch reads use the same function as before. Choose a site has a second mode that fills the destination box instead of opening a report. The print document leaves Tools out. The tests add a pretend destination site with an in-memory model, the reads the copy plan makes from the pretend source site (without changing any answer the existing tests use), and tests/tools.spec.js. The docs describe src/tools/, the write helper, the per-site digests, and the pretend destination, and the project terms add Tools, Copy site structure, and Destination. A reviewer must check by hand on a real tenant, with a destination site that can be thrown away: that Check destination is right for a site collection administrator and for someone who isn't one; that a copied list gets the chosen address and then the chosen name, with the destination's own digest; that every $select the copy plan sends is accepted; and that the screen works with a screen reader, at 200% zoom, and with ?csp=enforce and no Content Security Policy violations.
Copy site structure now copies the term sets the person chose from this site collection's own term group into the destination site collection's own term group. When the destination has no term group of its own, the copy creates one through CSOM with TermStore.GetSiteCollectionGroup, using TaxonomySession.GetTaxonomySession. It never creates or changes a tenant-level group. For each chosen term set, the copy uses a destination term set with the same name when there is one, and otherwise creates it. It then creates the terms with parents before children, with their synonyms, descriptions in the destination's languages, and custom properties. A term under two parents is created under the first and reused under the others. Deprecated terms and terms that are unavailable for tagging get those settings last. Terms already under the same parent with the same default label are used instead of being created again, so a second copy adds nothing twice. Labels in languages the destination's term store doesn't have are left out, with one note for each term set. Each change goes to the term store REST API (/_api/v2.1) first. When SharePoint refuses it with HTTP 403 or a message about OAuth or notAllowed, that change and every later one go through CSOM. When CSOM fails too, the chosen term sets that aren't copied yet are reported as failed and recorded as missing, so the columns step can skip the columns that use them. Term sets the person left out are recorded as missing as well. Every term that couldn't be copied, or was copied only in part, has its own row in the results, with what happened and what to do next. The module also adds three helpers for the columns step: one that connects a managed metadata column to its term set through CSOM, one that finds the destination's TaxonomyHiddenList, and one that checks that a tenant-level term set can be found from the destination. A maintainer must check these on a real tenant with a site collection administrator who isn't a term store admin: that the CSOM requests with GetTaxonomySession work and GetSiteCollectionGroup creates the group; that the REST writes with browser cookies are accepted or refused in the way the copy expects; which term a reuse relation's address names; that the isOpen, isAvailableForTagging, and isDeprecated changes are accepted; the form of parentSiteId in the group list; and that setting SspId, TermSetId, and AnchorId on a column reached through Fields.GetById binds it without a cast.
Copy site structure now copies each chosen list or library with its settings, columns, site columns, content types, views, and form settings. Columns are made in passes across every list so each one can find what it uses: plain columns, then lookup columns (after every list exists, pointed at the destination's list and site), then managed metadata columns (after the term sets, with the hidden Note column first, pointed at the destination's term store and TaxonomyHiddenList, then connected to the term set and read back), then calculated columns in the order their formulas use each other, then the columns a lookup column shows, made with adddependentlookupfield. Column formatting, the Title column, and the list's validation formula follow. The default view is updated in place, other public views are made with views/add with their layout and formatting, personal views are skipped, and hidden views are counted in one row per list. Site content types keep their source IDs, parents first, with their columns added through CSOM. A chosen list or library whose address matches one the destination already has with the same template, such as Documents, is no longer an error. The address box says the copy adds what is missing, and the copy uses that list: it adds the columns and views it doesn't have and leaves the rest alone. A match with another template is still an error. Failures are handled one item at a time. Each column, view, content type, and form gets its own result with a note that says what happened and what to do, and a stop of the copy is passed on. A managed metadata column whose term set wasn't copied, or that the destination's term store doesn't have, is skipped with a note naming the column, the list, and the term set. Check these by hand on a real tenant, because the pretend site only models them: whether createfieldasxml keeps internal names with Options 8; which formula form SharePoint accepts for calculated columns; that the CSOM requests for content type columns (FieldLinks.Add) and for connecting managed metadata columns work as written; that views/add accepts ViewType2, ViewData, and CustomFormatter and that MERGE on the default view accepts ViewType2; and the UniqueContentTypeOrder body.
Copy site structure now copies the files it promises to copy. Site Assets is copied folder by folder, with empty folders, in path order, leaving out the library's Forms folder. The destination gets a Site Assets library when it has none. Agents (.agent files) are copied to the same place in the matching destination library, which is the library the copy made from theirs or a library at the same address; each copied agent's note says it still uses the original site's knowledge sources. Site skills are copied folder by folder to Skills/<name>/ in the destination's Agent Assets library, never the Plans folder. When Agent Assets is missing, the copy turns on its feature and looks again, and skips every skill with a note when it is still missing. Modern pages are made again through the site pages API (create, save, publish), with this site's address, list IDs, view IDs, file IDs, site ID, and web ID in their content changed to the destination's. A page that still holds an ID of this site with no match in the destination is marked Partly copied. Pages in Templates are saved as page templates. Classic pages and pages in other folders are copied as files. The home page becomes the destination's home page only when Replace is checked. Each file is first copied on the server with SP.MoveCopyUtil. CopyFileByPath. When SharePoint answers that with a 4xx other than 429, the file is downloaded and uploaded with Files/AddUsingPath. Files over 250 MB are skipped with a note, and files and pages the destination already has are skipped unless Replace is checked. Each file gets its own result, and a failure doesn't stop the others. A page that fails after it was checked out is discarded, and a page the copy made is moved to the recycle bin, so nothing is left checked out. Every address uses parameter aliases, so quotes, # and % in names are safe. Check these by hand on a real tenant, because they rest on community sources or are untested: CopyFileByPath between two site collections and its 403 fallback; parameter aliases in Folders/AddUsingPath and Files/AddUsingPath; the folder read with $expand=Folders,Files for a folder that doesn't exist; turning on the AgentAssets feature by REST; whether a new page comes back checked out; SavePageAsTemplate, the name it gives the template, and removing the page it was made from; and which web part settings real pages hold, such as the List and Image web parts.
When the Permissions switch is on, Copy site structure now copies the permissions of the site into the destination. It maps built-in permission levels by their type and reports any whose permissions differ, without changing them. It maps custom levels by name or creates them, and when SharePoint refuses a new level because one with the same permissions exists under another name, it uses that level. The owners, members, and visitors groups map to the destination's own groups by their role. Other groups that give access to the site or to a copied list match by name or are created with their settings, then get their owner (a person through SetUserAsOwner, a group through a CSOM request) and their members. It gives the same access on the site, and gives each copied list with its own permissions the same permissions. People picker columns limited to a group are pointed at the destination's group. The copy never removes access or members that the destination already has. The only exceptions are the access SharePoint gives the person running the copy on a new group or list, which is removed when that person doesn't have it in the source. Access and group members that only the destination has are listed in a note, by display name only. It leaves out and reports sharing links, the Limited Access System Group, Limited Access and hidden levels, apps, the source site's own Microsoft 365 group (every Microsoft 365 group when that can't be read), and members who no longer exist. Everyone and Everyone except external users are copied as partly copied, with a note that they open the item to the whole organization. For a Teams channel site destination it changes nothing and says that Teams manages those permissions. Each item is logged with its own result and the copy keeps going after a failure; only a stop ends it. Notes never contain sign-in names or email addresses: a name that looks like one is replaced, and SharePoint's own messages are cleaned of them before they are quoted. New destination checks read back permission levels, groups, group members, access, and lists with their own permissions. Check by hand on a real tenant: - SetUserAsOwner and the CSOM request that makes a group another group's owner, which only community sources describe. - Whether creating a group adds the person running the copy as a member, and which access SharePoint keeps for them after breakroleinheritance(false,true) on a new list. - The error a person who no longer exists gives when added to a group or through ensureuser. - A subsite destination that uses its parent's permissions: custom levels are created on the top site, and the subsite gets its own permissions with breakroleinheritance(true,false). - The SelectionGroup MERGE on people picker columns, which the tests can't reach until the list column step fills the column map.
Copy site structure can now start from a data dictionary file (.xlsx) in the format in docs/data-dictionary.md instead of this site's report. The file is read in the browser and turned into the same copy plan that the site report gives, so the copy steps work the same way for both. From the file, the plan gets each list and library on the Assets sheet, its columns from the Fields sheet, choices and term sets from Value_Sets and Value_Set_Definitions, and lookups from Relationships. Each column's definition is built through the DOM, never from joined text. Columns of several types are supported: text, multiple lines of text, number, currency, date, yes or no, choice, lookup, person, hyperlink, managed metadata (with its hidden note column), and calculated. Files from other tools that give only display_type values are read too. The term sets of the site's own term group are copied with their terms, parents, synonyms, and descriptions; term sets the file lists no terms of keep their IDs and are never copied. Views, content types, pages, Site Assets, agents, site skills, and permissions aren't in the file, so they aren't copied. The file is refused, with a message that says what to do, when it isn't an .xlsx workbook, is larger than 25 MB, can't be read, has no Assets or Fields sheet, uses format 2.0 or later, has no lists or libraries, or holds a hidden-value placeholder such as [phone number hidden] in a name or choice, or anywhere when the file says values were hidden. Columns the file can't describe, such as a lookup whose list isn't in the file or that has no field ID, are left out with a note before the copy starts that says what to do instead, and are listed for the results table. A column from another tool whose name SharePoint keeps for its own columns, such as Order, gets a new internal name, such as Order2, and is still made. Any unexpected problem while reading the file shows the message for a file that can't be read. To check by hand on a real tenant: save a data dictionary with Show hidden values on, choose it in Tools, and copy to a test site. Then check that lookup columns show the Title column, that managed metadata columns are connected to the copied term sets, that calculated formulas were accepted, and that internal names match the source.
The five parts of Copy site structure (term sets, lists, files, permissions,
and data dictionary files) each passed their own tests, but 11 tests failed
when they ran together. This change fixes the causes and adds tests that run
whole copies.
Fixes for the tests that failed together:
- The pretend destination answered every CSOM request with the first part's
answer that matched. The lists answer took the request that connects a
managed metadata column to its term set, which taxonomy.js sends, and saved
its settings as text. The lists answer now handles only content type column
requests, and the taxonomy answer handles the connection for list and site
columns and updates the column's definition, as SharePoint does.
- The permissions tests counted every CSOM request and every change to the
list. They now count only the permission requests, because the term set and
column steps now send their own.
- The pretend source site didn't answer two reads the file steps make: the
site skill's folder and the site's home page. Both are standard answers now.
- tools.spec.js expected the stub results ("isn't available yet") of the term
set, column, view, Site Assets, and page steps, and expected the term set
step to send nothing. It now checks the real results, that the term set
changes come before the list, and that Documents is used as an existing
library instead of marked as a problem.
- The data dictionary test expected no row for the Title column, which the
lists step now renames to match the source.
Shared changes requested by the module builds:
- runner.js: checks each list with snaToolsListChoiceProblems, so a copy into
a library the destination already has no longer stops before it starts;
adds the Skipped rows of a data dictionary file with
snaToolsLogWorkbookSkipped; adds fileUrls and fileIds to the maps.
- plan.js: recomputes a data dictionary file's warnings when the person
chooses lists and term sets, so a lookup to a list they left out is named
before the copy; describes lookupFieldGuessed and skippedItems.
- ui.js: counts a managed metadata column once, without its hidden Note
column.
- lists.js: adds a note to a lookup from a data dictionary file that shows
the Title column; when permissions are copied, a Person column limited to
a group is no longer marked as partly copied, because the permissions step
sets its group.
- taxonomy.js: connects a site column to its term set too.
New tests in tests/tools-integration.spec.js copy a site with two lists, a
lookup, a managed metadata column, a calculated column, a formatted view,
Site Assets, a page, an agent, a site skill, and permissions. They check the
order of the changes, the results, and the check afterwards, including an
item the destination doesn't have afterwards. Two more tests copy with
Permissions off and from a data dictionary file. docs/development.md
describes the new tests and how the parts share CSOM requests.
A CSOM error about one term, such as a label SharePoint doesn't accept, no longer stops the rest of the term store copy. Only an access denied answer, HTTP 401 or 403, or a "Cannot find stub for type" answer before CSOM has worked once marks the term store as blocked. A reused term is made unavailable for tagging only when it is unavailable in the term set being copied, not in another term set it also appears in. When the REST API refuses a new term set's open or tagging setting, the copy now sends that setting through CSOM and uses CSOM for the rest of the copy, as it does for every other refused change. Term set notes now show language names, such as French (France), instead of language codes, and the note about a term set that couldn't be read in full says what to do next. The error for a value that can't be sent now says what to do instead of describing the code.
A failed read of one list's access in this site no longer stops the whole Permissions step. That list keeps the site's permissions in the destination, and its row says how to give it its own by hand. When a list gets its own permissions, the copy now reports access that SharePoint kept on it and that this site doesn't give, such as a group the person running the copy is in. The list's row is marked Partly copied when some people or groups didn't get their access, or when the copy couldn't read the list's access afterwards to check. Groups that a copied people picker column is limited to are now copied even when they give no access, so the column points at the same group in the destination. A new group's owner follows the same rules as members and access: this site's Microsoft 365 group, apps, and SharePoint's own account are left out, and the note says how to choose an owner. The note for a built-in permission level the destination lacks now gives a step the person can take, and the note for missing levels handles one level and several.
Add snaToolsRewriteSourceLinks to core.js. It changes links to the source site in every stored form (absolute, server-relative, URL-encoded once or twice, JSON-escaped, and HTML-escaped) and source list, view, column, term, file, site, and site collection IDs in every form. It keeps boundaries, so look-alike sites and other hosts stay as they are. Lists copied under a new address get the new address. Subsites, sharing links, IDs with no match, and lists the copy didn't make are returned as unmapped, so the page is marked Partly copied. Navigation can use the same helper later. Copied pages now use it in place of the local rewrite in files.js. A modern page in a folder is copied on the server, and then its links are updated through the site pages API. It is never uploaded, because an upload leaves the page blank. Fixes from the review of the files module: - Copy page templates before other pages, so a page with the same name doesn't block its template. - Send IF-MATCH and X-HTTP-Method with SavePageAsTemplate and SavePageAsDraft, as the CLI for Microsoft 365 does. - Copy an agent saved in Site Assets into the destination's Site Assets, and make that library when the destination has none. - Add files the destination already has to the file maps, so pages point at them. - A Site Assets folder that can't be read fails on its own row, and a failed read of the destination's lists fails only the agents, not the site skills. Add tests/tools-links.spec.js with the cases from the link research and its review, and tests in tools-files.spec.js for each fix, including a home page with many link forms.
Copy site structure now copies the source site's own content types, whether or not a chosen list uses them. The Tools screen has a Content types part with a checkbox for each one, all checked at first. A content type that a chosen list uses, or that another chosen content type is based on, stays checked and says why. A data dictionary file has no content types, and the part says so. Each copied content type gets its columns through CSOM with the same Required, Hidden, and display name settings as the source, such as Enterprise Keywords shown as Tags. Site columns that only a content type uses are copied first. SharePoint's own site columns that were renamed in the source get the same name in the destination, with a result row of their own. Content types from the content type hub are reported with what to do instead of being made as local copies, and so are content types based on them. The review fixes: - A view named All Items is now made when the source default view has another name. - A list the destination already had keeps the formatting of its built-in columns. - SharePoint's own site columns that someone added to a list, such as Start Date, are added from the destination's site columns. - A new list keeps its Item content type when its own content type couldn't be added. - Views leave a skipped column out of their sorting, grouping, filters, and totals, and say so. - A site column that shows another column of a lookup gets a Skipped row. - The note for a list the destination already had says what happened.
When the REST API refused a new term set's settings and CSOM refused them too, the copy marked the term store as blocked. Every later term and term set then failed without being sent, even though the REST API had just created a term set. The same happened when the REST API and CSOM both refused one term's settings. Now the term store is marked as blocked only when CSOM is refused before CSOM or the REST API has worked in this copy. Otherwise only that change fails, and later changes go through the REST API again. A new term set's settings switch the copy to CSOM only when CSOM makes them. The note for a term set that isn't complete now says the data dictionary file doesn't list every term when the copy comes from a file, instead of blaming a site that doesn't exist.
When a list gets its own permissions, SharePoint keeps access for the person running the copy or for a group they are in. If this site gives that person or group less access, the extra permission levels were never reported. Each person or group's row now names the levels they also have in the destination that this site doesn't give them, and the row is marked Partly copied. The same check runs for access to the site. For the person running the copy on a list, the copy removes those extra levels when it gave them every level this site gives them, and otherwise reports them. Notes about a new group's owner now say "the same owner" when the owner has no display name, instead of sentences such as "with its owner as its owner". A people picker column limited to a group that no longer exists in this site now says so and what to do. The note for an owner left out because the site's Microsoft 365 group couldn't be read no longer repeats "Sitenalyzer". The architecture page no longer says that the Tools files for data dictionary files, term sets, files, and permissions are not built yet.
SharePoint's own page and media content types are now recognized by their full IDs. A content type someone based on Site Page, Rich Media Asset, or a publishing page used to count as SharePoint's own, so the Tools screen didn't offer it and the copy dropped it without a note. A list content type that comes from SharePoint, such as Event, now gets a Skipped row when the destination list doesn't have it. The copy matches these by ID as well as name, so Item under a name in another language is kept and not removed from a new list. Folder and hidden content types aren't reported. The hidden Note column of a managed metadata site column is no longer named in a content type's note. When the copy makes the site column, the content type gets the Note column too; when the destination already has the site column, the copy uses the destination's Note column with the same ID or internal name. When a new content type's columns can't be read in the destination, the copy now uses the columns of SharePoint's own content type it's based on, so SharePoint site columns such as Enterprise Keywords are still added. When that parent isn't known either, the note names the columns to check. Only SharePoint site columns that someone added, such as Enterprise Keywords or Start Date, get their source names in the destination. Core columns such as Title, and columns a content type gets from its parent, keep the destination's names, so a source site in another language no longer renames them across the destination's site collection.
Copy site structure now reads the source site's subsites before it copies pages, so a link into a subsite stays as it is and the page is marked as partly copied, instead of pointing at a destination subsite that doesn't exist. When the source is a subsite, the site collection ID in page content is changed only next to the source site's own ID. A web part that points at the parent site keeps its site collection, so it still opens the parent site's content, and the page is marked as partly copied. Sitenalyzer finds out whether the source is a subsite by reading the site collection's top site. When that read fails, it treats the source as a subsite, which never leaves a web part pointing at nothing. A page template whose temporary page couldn't be removed no longer makes the real page with the same name say the destination already has it. That page now fails with a note that says to delete the temporary page and copy again. The note for a site skill now names a folder that couldn't be read apart from the files that couldn't be copied, instead of counting the folder as one file. The note for an agent whose destination lists couldn't be read now gives advice that fits the error. A page in a folder, or a classic page, that replaced one in the destination now says it was replaced. The pretend source site in the tests answers the two new reads. The architecture and development pages now describe files.js, the link rewrite in core.js, and tools-links.spec.js.
Copy site structure has two new switches under Other content. Navigation is on by default for a site source. It copies the left menu, the top menu of a classic site that isn't a hub, and the footer's links. Links stay in order with their labels, hidden links, and audiences, and audience targeting is turned on in the destination when this site has it on. Every link into the source site is pointed at the destination through snaToolsRewriteSourceLinks, and links to other sites stay the same. Each menu is made the same as the source's: the destination's own links are removed only after the source menu was read, so a second copy adds no link twice. A hub menu is reported, not copied. Site look is off by default. It copies the theme colors with ApplyTheme, the header, menu style, and footer settings with SetChromeOptions, whether the site navigation shows, the site logo and its thumbnail (the group picture on a site with a Microsoft 365 group), and the footer's display name and logo. Pictures are copied into the destination's Site Assets first. Each setting gets its own result row, and parts SharePoint refuses because custom scripts are off or a SharePoint admin is needed are skipped with what to do instead. Both switches are disabled for a data dictionary file. The new code is in src/tools/site.js, built after files.js. The runner copies the site look after Site Assets and the menus after Site pages, both before Permissions, and the check afterwards reads both back. New tests are in tools-site.spec.js with a pretend destination for menus and look. The integration test now copies a left menu too, and the tools.spec copy with the other parts off also turns Navigation off.
When the destination already had a list with its own permissions, such as on a second copy into the same site, the permissions copy removed levels from the person running the copy, and their whole access, even though they had that access before the copy. The notes then said SharePoint gave them that access. The copy now reads whether each destination list already has its own permissions before it changes anything. It removes access only from a list that used the site's permissions, because only then did SharePoint give that access during the copy. A list that already had its own permissions is left as it is and only gets the access this site gives. When the copy can't read this, it removes nothing and says what to check. Lists now get their own permissions without clearing the permissions of their items, so a list the destination already had keeps them. The note about a level that a person has only in the destination now says "they" instead of "it" for a person. The destination check replaced the whole note of a permission row when it found a missing level, which erased the notes about extra levels and about opening the site to the whole organization. Those sentences are now kept after the check's own note.
When Sitenalyzer couldn't read a new site content type's columns in the destination, it used the parent's columns only when the parent was one of SharePoint's own or was in the copy. A content type based on a content type hub type, such as Policy Document, got a note that named every column it has, including hidden ones such as Source Url and the Content Type column, which people can't see in the content type's settings. The plan now keeps the columns of every parent the source site lists, so the copy uses them for hub parents too. When the parent isn't known, the note leaves out hidden SharePoint site columns and the Content Type column. The Skipped note for a SharePoint content type the destination list lacks now says what happened: the destination list doesn't have it, and Sitenalyzer doesn't add those. The old note gave a circular reason.
When the source is a subsite, Copy site structure now finds the site collection ID in page content in every form the link rewrite finds it, such as encoded twice or with %252D between its parts. Before, those forms were missed, and the page said Copied while it still held the source's ID. The page is now marked as partly copied while the old ID is still in it in any form, found by its 32 hex digits. The part of the text checked for the destination site's own ID now ends at an HTML tag edge, and for an ID in a link, such as ?siteid=, at the end of that link. So a link to the parent site that follows a link to the subsite itself keeps its site collection ID instead of getting the destination's. Braces right around any ID, such as the list ID of a web part item, no longer count as the edge of that part, so an item that lists its site ID last is changed as it should be. The GUID helpers of the link rewrite are now shared functions in core.js, so the files copy uses the same rules. The note for a page blocked by a page template's temporary page is split into shorter sentences.
Copy site structure could leave menus and the site look in a different state than its results said. This change fixes those cases and reports the ones SharePoint doesn't allow. Menus: - A link added with CSOM after the REST API refused it is now hidden and given its audiences with a change that leaves out its title and address, which SharePoint could refuse again. If SharePoint still refuses, the link is reported as partly copied and showing to everyone, instead of failed. - The save through the menu editor now sets each copied link's hidden state as well, and reads the menu back to confirm it, so a hidden link can't become visible without a row saying so. - When this site has audience targeting off, links are copied without their audiences, so they still show to everyone. All copied menus are checked for audiences, not only the left menu. - Links to the group's notebook, Teams, and Planner stay as they are and are reported as still pointing to the old group. Links kept as they are, such as # or SitePages/Home.aspx, now say so instead of claiming to point outside this site. - Top menu and footer links that open in a new tab are reported, because SharePoint keeps that setting only for the left menu. - A removal that stops partway says how many links are gone. The row about removed links is no longer counted as a copied item. - A menu with more than 500 links is reported instead of being copied short without a word. A footer that couldn't be read is reported apart from one SharePoint didn't let Sitenalyzer set up. Site look: - Footer settings are copied only when this site also has a footer, so a team site no longer turns off a communication site's footer. - When the destination's header and footer settings can't be read, nothing is sent, because SharePoint changes them all at once. - A group site's logo is reported as copied when only reading it back failed, and the copy goes on to the thumbnail. - When SharePoint refuses to change a group's picture, the logo is saved as __siteIcon__.png in Site Assets, which Microsoft documents as the way around that refusal. - The theme row says that the colors of page sections might differ, and the Site look hint is one sentence.
Copy to destination now runs a read-only check of the destination before the copy starts, so people learn about failures they can predict, such as term sets their organization won't let them create, before anything changes. The checks live in the new src/tools/preflight.js, which the build places after verify.js in both build.sh and build.ps1. Each check is added with snaToolsAddPreflightCheck and returns warnings that say the item will fail, will be skipped, may fail, or are a note. The checks cover term sets the destination can't take (naming every column that uses them), columns whose term set or lookup list was left out, calculated columns that use a skipped column, term sets of the organization missing from the destination's term store, content types from the content type hub, a missing Agent Assets library, Site Assets files over 250 MB, classic pages and site look settings on a destination without custom scripts, a Teams channel site with Permissions on, and a list address the destination now uses for another kind of list. Reads go through the existing read helpers; the one CSOM request only loads the term store and is refused by its sender if it could change anything. The window.confirm prompt is replaced by a "Before you copy" part of the Tools screen with a summary, a table of warnings, and Copy anyway and Go back buttons. After the copy, the warnings appear at the top of the results table with the result Warning, and the counts sentence mentions them without counting them as copied, skipped, or failed. The lookup and term set warnings that plan.js used to add are now part of the check. Tests: tools-preflight.spec.js covers each check, Copy anyway, Go back, a check that fails, keyboard use, and that the check sends no change. The tests that accepted the old confirmation now select Copy anyway.
Add a task page, docs/how-to/copy-a-site.md, that walks through Copy site structure: who can use it, choosing a source and a destination, the lists and libraries table with its Name and Address boxes, term sets, content types, each switch under Other content, Replace, Stop copying, the results table, the destination check after a copy, and copying again into the same destination. Update the other pages to match the code as it is now. usage.md gets a Tools section with the #sna/tools and #sna/tools/copy addresses. architecture.md adds site.js to the file table, the requests that Tools sends when it opens and for Navigation and Site look, and the full copy order. design-decisions.md explains the main choices, such as the copy order, creating lists with the address first, Permissions off by default, and the few cases where the copy removes something. troubleshooting.md lists six problems the copy reports, each with what to do. data-dictionary.md describes how a data dictionary file is used as the source, how lookups are matched, and which files are refused. Also add the page to the docs index and the README, add site.js to the build order in development.md, and add the Tools terms for site collection administrator, SharePoint admin, Navigation, Site look, Replace, results, and the destination check to the project terms.
When the destination already had a list that the copy used, and that list used the site's permissions, the copy gave it its own permissions with only the access of the person running the copy. Everyone else lost access to it, such as the destination's members and visitors groups on the Documents library. The lists step now records each list it used instead of creating, and the permissions step gives such a list its own permissions with a copy of the site's access. It then reports the access this site doesn't give instead of removing any of it, including the access of the person running the copy. The row for a list that already had its own permissions no longer says the access was added before any was given. It now says only that no access was removed, and the rows for each person or group say what was given. The Other access row no longer names the person running the copy in the third person. On a list, the Permissions row tells them about their own access. On the site, the Other access row does. When the row names one person, it now says "they" instead of "it". The guide and the design decisions page now describe how the copy treats a list the destination already had.
When the destination already had a list that uses the site's permissions, the copy gave that list its own permissions only after it had given people access to the destination site. The list then got everyone the copy had just added to the site, such as Everyone except external users, even when this site limits the list to a few groups. The results said this access was already there, so the person had no reason to remove it. The permissions step now gives each such list its own permissions before it gives anyone access to the site. The list keeps the access people had to it before the copy, and nobody gets access to it only because the copy gave them access to the site. The list's row now says it kept the access anyone had to it before the copy. The test for this case now checks that the groups the copy adds to the site get no access to the list, and that the members group keeps only the level it had before the copy. The guide and the design decisions page now describe the new order.
On a subsite, the site's own content types leave out the ones it gets from the sites above it, such as Document. When a copied content type was based on one of those and the destination couldn't list the new content type's columns, the copy didn't know which columns came from the parent. Its note then named the parent's hidden columns, such as _SourceUrl and TemplateUrl, as columns to check. The copy plan now reads the site's available content types when a copied content type's parent isn't among the site's own, and uses that parent's columns. When that read fails too, the copy works as before and names the columns it couldn't check. A new test copies a subsite content type based on Document while the destination refuses to list its columns, and checks that Start Date is added with no note. The existing test for the note now refuses the available content types read, so it still reaches the case where the parent is unknown. The architecture page lists the new read.
When the source is a subsite, Copy site structure changes the site collection ID in a page only where the subsite's own ID is in the same part of the text. For an ID that follows = in a link, that part ended at the nearest space or quote, so a page with an item such as a Quick links entry with siteId, webId, and a link ending in ?siteid=<id> was marked Partly copied even though the item named the subsite. The shorter part is now used only when the link itself names a site with webid= or guidWeb=. Otherwise the whole item or HTML tag is checked, as before. A link now also ends at a quote written as a code, such as \u0022, %22, or %2522. Before, a link to the parent site that followed a link to the subsite in that form got the destination's site collection ID while keeping the parent's site ID. Two tests cover these cases, and both fail without the change.
…e item When the source is a subsite, Copy site structure changes the site collection ID in a page only where the subsite's own ID is in the same part of the text. For an ID that follows = in a link, the part ended at the nearest space or quote, and when that part didn't name a site, the whole web part item was checked instead. A query such as SiteId=<id> WebId=<parent site's ID> in an item that also held the subsite's ID therefore got the destination's site collection ID while still naming the parent site, so the query pointed at a site that doesn't exist and the page was marked Copied. Now the text up to the nearest quote is checked before the whole item. A query that names the parent site keeps its site collection ID, and the page is marked Partly copied. A query that names the subsite still changes, and a lone ?siteid=<id> link still uses the whole item. A new test covers this case and fails without the change.
A hidden link with audiences that the REST API refused was added with CSOM and then changed once, with both its hidden state and its audiences. A tenant that refuses audiences refused the whole change, so the link stayed visible. The copy now hides the link and gives it its audiences in two separate changes. The left menu's read-back checks the two apart, and the link's row and destination check follow what SharePoint confirmed. When SharePoint refuses to change a group's picture, the copy saves the logo as __siteIcon__.png in Site Assets. It now does this only for a PNG logo, and never replaces a __siteIcon__.png the destination already has. In that case the row says how to change the site logo thumbnail by hand. A group site's logo whose read-back failed no longer gets a destination check, because changing the group's picture afterwards can change the logo. When the communication site feature can't be read, the copy no longer treats the site as one without a footer. For this site, the footer links are still copied, and the footer settings row says Sitenalyzer couldn't check. For the destination, the footer and the menu orientation are left alone, with rows that say why. When this site's settings can't be read, the row about audiences now says Sitenalyzer couldn't check audience targeting, instead of saying it is off. The group notebook check now matches only a Site Assets folder whose name ends in " Notebook", with or without a slash at the end, and the note for these links uses one name for the group.
When SharePoint refuses to change a group's picture, the copy saves a PNG logo as __siteIcon__.png in Site Assets, and never replaces a file that is already there. On a second copy, that file is the one the first copy saved, but the row still said the thumbnail was skipped and asked the person to upload the copied logo by hand. The copy now reads the existing __siteIcon__.png and compares it with the copied logo. When they match, the row says Copied and that Site Assets already has the copied logo under that name. When they differ, or the file can't be read, the row keeps the note that says how to change the thumbnail by hand. New tests cover a second copy and a file that can't be read.
The check before copying now matches what the copy does, so Before you copy no longer says the copy is clear when it will fail or stop. - When the destination already has its own term group, the term set check still reads the term store through CSOM and warns that changing term sets may be refused, because a group doesn't mean the person may change it. - The Lists and libraries check runs the same name and address checks as the copy, so a name the destination started using after Check destination is found. A problem that stops the whole copy is marked as such: Before you copy then says the copy can start only after the problem is fixed, offers only Go back, and Lists and libraries marks the box to change. - A classic page the destination already has isn't listed as failing when Replace is off, because the copy leaves that page as it is. - Managed metadata columns with no term set, and the columns a data dictionary file can't make, are listed as skipped, with the calculated columns that use them. The copy now skips those calculated columns too, instead of sending formulas that use missing columns. - Messages say "your term store administrator" everywhere. The destination check now refuses this site, by its ID when both IDs are known and by its address otherwise, because a copy into this site would change it. Replace is off and grayed out for a data dictionary file, which has no pages or files. A failed read of this site while the copy is prepared gets its own message instead of one about the destination. The unused SNA_TOOLS_NOT_READY_NOTE and a plan warnings call that did nothing for files are removed. Tests cover each change, including that the CSOM read refuses a request that could change something. The how-to, troubleshooting, architecture, design decisions, data dictionary, and usage pages describe the new behavior.
Check destination refused the site whose report is open even when the source was a data dictionary file. None of the reasons for that refusal apply to a file: navigation, site look, and permissions are off, and no list of this site is the source. Someone building the lists of a new, empty site from a file could not copy into it. Check destination and the check at the start of the copy now refuse this site only for a copy from this site's report. For a file copy into this site, the hint under Copy says the copy goes to this site instead of saying nothing changes in it. When a check found this site, or is still running, changing the source drops that check and asks the person to check the destination again. The design decisions and architecture pages now say that Check destination refuses this site, since "destination check" means the reading back after a copy. The how-to, troubleshooting, usage, and data dictionary pages describe when this site can be the destination. A new test copies a file into this site and checks that the same site is refused for this site's report.
Add a "Before you copy" section to the how-to page. It describes what the check reads, what will fail, will be skipped, and may fail mean, the Copy anyway and Go back buttons, a problem that stops the copy and leaves only Go back, checks that couldn't finish, and the Warning rows that the check adds to the top of the results. The usage and troubleshooting pages now describe the same steps, and the terms list adds "Before you copy" and the Warning result. Check every statement about Tools against the code and fix the ones that didn't match. The pages no longer describe a confirmation question before the copy. The destination messages are quoted in full, the data dictionary switches and the tests of tools.spec.js and tools-preflight.spec.js are described as they are, and the architecture page lists the reads that the check makes. The design decisions page no longer gives a reason for Permissions being off at first beyond what the switch does, and no longer gives reasons for the site collection administrator check or the read back after the copy that the code doesn't support.
A page that named a later page by its ID, such as in a Quick links web part, was saved with the source page's ID, because that page's destination ID didn't exist yet. After every page is copied, Sitenalyzer now saves such a page again with the destination ID and updates its result. A page still holding something of this site stays Partly copied, and a page SharePoint refuses to save again says so in its note. When the copy of Site Assets left the destination's own file at the logo's address, the Site logo and Footer logo rows said Sitenalyzer copied this site's logo. Those rows now say Sitenalyzer used the picture the destination already had, and how to replace it.
When the destination is a subsite, the copy now checks whether each of its owners, members, and visitors groups is also one of those groups on its parent site or on the top site of its site collection. SharePoint groups belong to the whole site collection, so adding people to such a group would also give them access to those sites. The copy no longer adds anyone to such a group. It copies this site's group the way it copies other groups: to a group with the same name, which it makes when the destination doesn't have one. When Sitenalyzer can't read the other sites' groups, it treats every one of the destination's groups as shared. Before you copy lists each shared group, and the pretend destination can now act as a subsite of a parent site so the tests can cover this. The permissions step now limits only the Person columns that this copy made. A Person column that the destination already had keeps its own group, and its row says which group this site uses. When the permissions step stops early, such as for a channel site or after a failed read, each Person column the copy made gets a Partly copied row that says it lets people choose anyone and what to do. The column's first row now says that it lets people choose anyone until the groups are copied.
Notes that quote SharePoint's own error message now replace every sign-in name and email address with "the account", and hide other values that look like personal information. This happens in one place, snaToolsServerSaid in core.js, so the notes of every step are covered, and the Permissions step uses the same helper instead of its own copy. Saved results files also hide such values in the destination's name in the file name, as the data dictionary and permissions workbooks do. After Stop copying, the copy still adds rows until it ends, so a change that SharePoint finished after the stop appears in the results. A request whose request digest was still on its way when the person stopped is no longer sent. When a stop leaves an item partly made, its row is Partly copied and says what to do next: a list without its chosen name or settings, a blank or checked-out page or page template, or a menu with only some of the destination's links removed. Any other change that has no row yet gets a Partly copied row for the step that was running. The results then say that the item being made is marked Partly copied. The destination check keeps the sentence about the hub when it finds other theme colors in a destination connected to a hub, and its note for items it couldn't check reads correctly for one item and for several. The how-to, architecture, and development pages describe the new behavior.
A copied agent still uses the knowledge sources in this site. Its note now says so with the project's term for the source, and tells the person to edit the agent in the destination to use the destination's lists and libraries instead.
Copy site structure no longer replaces the New menu order of a list or library the destination already had. It used to set that order from the source list alone, which took the destination's own content types off the New menu. Now it adds the content types it copied to the end of the list's own order, and leaves a list without an order of its own as it is. When it has to turn on Allow management of content types for such a list, the list's row says so. SharePoint's own site columns, such as Enterprise Keywords shown as Tags, are renamed in the destination only when this site and the destination use the same language. When the languages differ, the column keeps its name and its row says why. The row for a renamed column now says that the new name shows in every site of the destination's site collection and how to undo it, and Before you copy lists each column the copy will rename. The note under the Address box of a list or library the destination already has, and a new row in Before you copy, now say when the copy will rename it to the name in the Name box. A lookup site column in a site content type is no longer reported as Failed in each list that uses the content type. Adding the column to the content type also adds it to those lists, so the copy now reads each list's columns after that, and the column's views and the columns that show another column of it keep it. The pretend destination in the tests now answers the New menu order of a list, the language of the top-level site, and adds a site content type's columns to the lists that use it, as SharePoint does. The how-to guide and design decisions describe the new behavior.
Content type names in Tools are now hidden like every other value the first time the Content types part is drawn, so a name that looks like a phone number no longer shows until a later redraw. Use this site in Choose a site now keeps the dialog open and says what to enter when the address box is empty, instead of clearing a destination that was already checked. Typing another address or choosing another site while a destination check runs now drops that check's answer, so it can't answer for an address that is no longer in the box, and Check destination works again right away. Empty status lines now stay on the page instead of being hidden, and the messages written after the view is drawn again (Go back, a source change, and a file that was read) are written a moment later, so screen readers announce them. The table of lists now becomes a keyboard-scrollable region whenever its box changes size, such as after zooming in, not only when it is first drawn. The headings under Copy site structure, Before you copy, and Results are now one level under the part heading. While a copy runs, Refresh report, Start over, and Change site are turned off, with a sentence under them that says why, and leaving or reloading the page makes the browser ask first. Each of them would otherwise stop the copy and clear its results without asking. The check before copying now counts a term store it couldn't read as a check that couldn't finish, reads the destination's content types again before it warns about content types from the content type hub, and no longer says the destination lacks the list of a lookup column from a data dictionary file, which it doesn't check. When the whole check fails and the person copies anyway, the results now start with a row that says so. The tests cover each change, including that Navigation and Site look are off and disabled for a data dictionary file. The how-to, architecture, and design decisions pages describe the new behavior.
When the copy doesn't know a lookup column's source list, as with a data dictionary file, it never looks in the destination for that list. The skip note now says only that the list isn't in this copy, instead of also saying the destination doesn't have it.
The files in dist/ are built from src/ with build.sh, so a site owner can upload them and test Copy site structure on a SharePoint site.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9f23b9a164
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
Comment on lines
+462
to
+463
| var options = snaToolsFilesLinkOptions(plan, dest, maps, fileState, sourceLists, destLists); | ||
| options.wholeLink = true; |
There was a problem hiding this comment.
When Navigation is enabled but Site pages is disabled, fileState.subsites remains empty: its only initialization from SharePoint happens in snaToolsCopyPages, which also returns before that initialization when the source has no pages. Consequently, navigation rewriting changes a link such as /sites/source/archive/SitePages/Home.aspx into /sites/destination/archive/SitePages/Home.aspx, reports it as copied, and loses the working link to the source subsite even though subsites are not copied. Load the source topology before constructing the navigation link options, independently of whether pages are copied.
Useful? React with 👍 / 👎.
Comment on lines
+855
to
+857
| var plan = await snaToolsReadWorkbook(file); | ||
| if (_snaToolsState !== state) { return; } | ||
| state.workbook = { fileName: file.name, plan: plan }; |
There was a problem hiding this comment.
If someone selects workbook A and then selects workbook B before A finishes reading, both requests retain the same state, so this check accepts both results. If A finishes last, it replaces B's copy plan and redraws the choices for the wrong file. This can cause the destination to receive a different structure than the person selected. Track a file-read generation or the selected file and discard superseded results in both the success and error paths, as the destination check already does.
Useful? React with 👍 / 👎.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request adds a Tools section to the report screen, with one tool, Copy site structure (#34). It copies a site's structure into another site where the person is a site collection administrator. It copies no list items and no library files, apart from site pages, Site Assets, and agents and site skills.
Merge order
This branch is built on top of #55, so its first six commits are the commits of #55.
mainand drop those six commits before merging. Nothing else changes.What it copies
Before and after the copy
Code layout
The new code is in
src/tools/.build.shandbuild.ps1join those files intositenalyzer.jsin a fixed order, so the app is still one script with no new globals. The last commit adds the rebuiltdist/, so you can upload it to a test site.Documentation is in the new
docs/how-to/copy-a-site.md, plus updates todocs/usage.md,docs/architecture.md,docs/design-decisions.md,docs/troubleshooting.md,docs/data-dictionary.md,docs/development.md, and the README.Testing
bash build.shandpwsh build.ps1write identical files.tests/pretend-*.js. It answers the REST and CSOM requests the copy sends, and it records every change so tests can check the destination afterwards.Checks to do by hand on a real site
?csp=enforceand check the browser console for Content Security Policy messages.Closes #34