UPDATE, note is below but I've confirmed this is a problem with ROS not confirming the saving of the custom default script. Previous versions it would prompt and ask to keep or revert, and that response was passed back to the script through $action. While it is setting $action="apply" before running the script, it's never actually confirming it within the OS somewhere, and thus causing the config to remain unconfirmed. I cannot figure out how to "confirm" to keep the default config via any script or scheduler.

Original Post for reference:
I'm having a strange issue with default configs that seems almost like either A) ROS is running the default config script run but thinking that it didn't complete for some reason (it does), and then at next reboot clearing the config and resetting everything again, and continuing to do this in a loop until someone goes in and makes some change to the config on the device (literally any change). or B) like following the default config script finishing, the running config isn't written to disk until a change is made by a user logged in.

I'm seeing this when testing on 7.11.2 and 7.12.2 on a hAP AC and a hAP AC3, have not tested this on other versions yet at the time of writing.

After performing a factory reset and applying a custom setup script that's loaded via branding maker the router applies the configuration as expected and without problem. The issue comes when I reboot the device. When rebooted, the config is usually lost and the default script is reapplied. This happens multiple times in a row, usually until someone (or a script) actually changes something in the config while it's running, and only then does it retain the config.

For testing I just loaded set reset two routers to default and loaded my defaults. Routers booted up fine and configured everything as normal. I rebooted them 5-6 times each and every time could see from the logs after logging in they had been reset and reapplied all the defaults again. It wasn't until I made a change to the config (as a test I changed the clock timezone on one and manually added a route on another) that it would retain the config and not reset every time it was rebooted. It's not just a software reboot, it's every time it reboots including power loss.

This may not seem like much of a problem, but our default setup script is paired with an inhouse provisioning tool, and most of these devices are NEVER logged into even a single time. We load the default configs on in bulk in the production lab and ship the devices out to customers, once in the field they plug them in and the router phones home and pulls its running config from our provisioning system which is working fine. For most of the lives of these devices no one ever logs into the router, our provisioning system pushes firmware and config updates to the devices automatically, so them resetting their config each reboot when no-one logs in is actually a REALLY big problem as it means every time it's powered off or rebooted the router has to go through and build a new config from scratch, which our provisioning system can't tell the difference between a router that "forgot" it's config and one that's been intentionally factory reset and therefore should have new credentials and Wi-Fi settings deployed to.

Has anyone else ran into something like this? or does anyone know if there's something with :global defconfMode, $action or anything similar that needs to be set after the default config script completes in order to tell ROS that the script was successful and to NOT reset itself again next boot?

Just after posting this I thought to run a different type of test. I added an additional test script to my default config, and then added to an existing scheduled script a command to remove the test script. After reboot (and each reset), 30 seconds later the scheduled script runs and deletes the test script. If I reboot the router it still performs a reset after the next reboot. However if I manually delete the same test script before the scheduled script is able to delete it I find an additional entry showing up in the logs "system, info: configuration changed, confirming default configuration" and from that point forward I can in fact reboot the router without it resetting each time.

SO, entirely programmatically via a script, how do I confirm the default configuration so it's retained? This wasn't an issue in v6.

opened a ticket SUP-151561 as well.

Hmm. I have defconf scripts, but I've never see anything like this. But $action is provided in a custom V7 defconf, and you AFAIK you do not have to confirm anything.

Now I have NOT tested this recently... So perhaps this has changed when they separated out the caps-man defconf script.

Also, I use an additional check if $action is NOT provided (nil/empty/nothing), it assumes $action = "apply" – so if it $action went missing, I might not know....

Let us know what support says... MT has changed stuff in recent builds WRT defconf....

Hmm. I have defconf scripts, but I've never see anything like this. But $action is provided in a custom V7 defconf, and you AFAIK you do not have to confirm anything.

Now I have NOT tested this recently... So perhaps this has changed when they separated out the caps-man defconf script.

Also, I use an additional check if $action is NOT provided (nil/empty/nothing), it assumes $action = "apply" – so if it $action went missing, I might not know....

Let us know what support says... MT has changed stuff in recent builds WRT defconf....

I did a lot more testing last night, and absolutely confirmed that when my default config script finishes, and absolutely regardless of what other actions any scripts take, the configuration remains unconfirmed by ROS and will perform another factory reset & reload of the default config at the next reboot, unless I log into the router AND make a change via winbox (or probably webfig or ssh etc as well). As soon as any change of any sort is made by a user (as opposed to by a script that was created by the default config & running automatically), the router logs an entry of something to the extent of "configuration change made, default configuration confirmed" and then after that the router is working normally in every way for the rest of the life of the OS (at least as far as my testing can confirm). However if you have a device that's entirely setup automatic via a custom default script, it will rerun the default setup script every time it's rebooted, until a change is eventually made that causes the OS to "confirm" the default configuration.

As for the $action, I am hoping that it is possible to do something like but so far everything I've tried to do has failed to trigger the configuration to be confirmed & saved. I've added additional scripts and triggered additional system changes (via scripts) during and after the default config, yet nothing at all seems able to get the router to confirm the config without actually logging in and performing some config action. I've also tried opening terminal and viewing the license agreement (initially pops up post reset with a want to see y/n prompt), I've set and deleted global variables, and I've sent additional config changes to the router post reset & default config via the scripts running (they are loaded with my default config), and nothing yet has been able to trigger the config confirmation.

Just heard from support, their suggestion was just to use 7.15+ as it now supports hiding the default configuration prompt (and no regards whatsoever to the giant and still not production ready change in wireless architecture starting in 7.13 that I'm a long way from being able to roll out to production, or the fact that 7.15 isn't even released as a pseudo stable release yet!)

However this got me thinking, and I suspect that the problem may in fact BE the 7.15 change to hide the default configuration prompt that has actually broken the branding maker config for older versions. I wonder if the addition of this option to the branding maker has caused the files to be broken when loading onto older versions now.

Unfortunately if I'm correct and this is the issue then I'm absolutely screwed if MikroTik doesn't fix the branding maker tool to work again with <7.15 configs as upgrading to anything past 7.12.2 is not an option at this time given the change to wireless package & drivers, and even if I could go to the latest "stable" 7.14, the fix still isn't even available there, and I'm both absolutely unwilling and also prohibited from loading a RC build onto customer equipment, yet also have a deadline that I'm not going to be able to meet without this working.

I'll say this publicly, again, MIKROTIK PLEASE OFFER REAL ENTERPRISE SUPPORT OPTIONS, I WILL PAY YOU FOR IT!

Well better docs on the mechanics of branding/default configuration be a good start. But exactly how stuff like $action in defconf is suppose to work be good to document. Just not seeing enterprise support, if they cannot keep the docs up to date.

I have a lot of the wAPacRs with 16MB & use zerotier with branding kit custom defconf... Although have NOT gotten to problems with defconf, since everything past 7.12.1 is fraught with potential gotchas because of the package sizes.

But in same boat here. Does seem like they have made a lot of progress in 7.15. I think patience and sticking with 7.12.1 or prior is kinda only option at this point.

As for the $action, I am hoping that it is possible to do something like

Code: Select all

:set action "confirmed"

It may need at ":return 1" or something, but "guess-and-test" is rather annoying approach to something like this.

To be honest, I don't care how it works.

It just be good to know the right way in 7.15+ to deal custom defconf's – since I've never been sure how $action works other than testing it.

I've got an idea I feel is close to working & solving the problem for now, but I'm struggling to get over the finish line...

My idea is to use fetch to post to the local rest api on the device (credentials are already available in my script so no issues there) and change a setting somewhere. That should trick ROS into seeing a config change made by the user, and therefore confirm the config. And I think I've found an absolutely benign setting to change, /system note.

I've gotten the fetch tool to successfully login and perform a get, but for the life of me I can't get the format right for it to change a value. First time trying to use the new rest api, all our automation uses the original API.
The problem is every time I run it (also same when testing from my normal API testing tools) I get a response of

Code: Select all

tool/fetch url=http://192.168.88.1/rest/system/note user=user password=password http-method=post http-data="{\"note\":\"system configured\"}" http-header-field="Content-Type:application/json"

With POST, it URL is url=h ttp://192.168.88.1/rest/system/note/set

For a method=put, that might work with existing URL. I normally just use POST since the URL mimics CLI.

Code: Select all

tool/fetch url=http://192.168.88.1/rest/system/note user=user password=password http-method=post http-data="{\"note\":\"system configured\"}" http-header-field="Content-Type:application/json"

With POST, it URL is url=h ttp://192.168.88.1/rest/system/note/set

For a method=put, that might work with existing URL. I normally just use POST since the URL mimics CLI.

Yup, figured that out, and just finished testing and verified that the post to the API it does in fact confirm the config and eliminate the reset loop cycle. it's a bit of a weird solution, but it 100% resolves the issue.

Nice piece of troubleshooting and finding a workaround !!

verified that the post to the API it does in fact confirm the config and eliminate the reset loop cycle. it's a bit of a weird solution, but it 100% resolves the issue.

Great work. But this is a bug (or at least doc issue on how one should do this "correctly").

verified that the post to the API it does in fact confirm the config and eliminate the reset loop cycle. it's a bit of a weird solution, but it 100% resolves the issue.

Great work. But this is a bug (or at least doc issue on how one should do this "correctly").

100% yes, and according to support it's going to be addressed in 7.15+ (which for most of us really means the soonest we'll see it is 7.16+ because if you're not ready to jump on 7.15 already, you're probably planning to wait for a very stable next jump past 7.12 and let all these changes settle down), but for anyone trying to support custom default config scripts for anything before v7.15 this is probably going to be your only solution. The router won't confirm the config by any action taken launched from within the context of the default config script (at least none I tried, and I tried A LOT), so you have to reach outside the script context via fetch and authenticated in via the Rest API to trigger the router into accepting the config and not re-running the default setup script every single reboot.

Just heard back from support, they say there's no official fix coming, except for in 7.15. So if you are going to load default configs <7.15 (I saw the problem on 7.11 too, not sure when it started), this is your only solution.