Optimizing Neural Network Structures with Keras-Tuner

[@bhaskersriharshasuri7359]

"A paper bag can solve MNIST ". That should be a quote on a T-shirt.

[@AD-bz2ci]

I would buy it. Please make.

[@sparshgupta2931]

What does that mean??

[@asdfasdfuhf]

@@sparshgupta2931 That basically anything can be trained to recognize numbers from the dataset named MNIST: www.google.com/search?q=what+is+mnist&oq=what+is+mnist+&aqs=chrome..69i57.2387j0j7&sourceid=chrome&ie=UTF-8

[@Quitoss]

I’m a paper bag

[@deojeetsarkar2006]

Thanks for everything sentdex, your name'll always find a folder in my PC.

[@JordanMetroidManiac]

26:30 The number of possible hyperparameter combinations with that search space is 8^2 + 8^3 + 8^4 + 8^5 = 37440. So, of course, a random search on that could take up to 37440 trials to find the best possible combination of hyperparameters. There are usually subsets of combinations that are "alike" and would achieve similar performance, so you wouldn't need to set max_trials = 37400, but more like max_trials = 100.

[@abrahamowos]

I actually came to the comment session to find this. Thank you for posting this.

[@hassenzarroug9159]

Seriously man, you are the reason why i love machine learning! you make it look easy and fun which is the exact opposit of what my teachers are doing! Thank you so much and God bless you!

[@Accarvd]

Probably one of the best KZbin videos (on this topic)

[@ajaysingh8887]

Finally, this is what I was looking for.

[@Neogohan1]

Both Kite AND Keras Tuner were things I've been wanting for awhile as part of learning TF, and you managed to knock em both out in the one vid. Very useful stuff! Thanks!

[@jumpthecagemma4987]

Last comment - this tuner only works if calling keras directly from tensorflow Example: tf.keras.add.layers(etcc) Calling Keras on it's own provides an error about compiling a model Bad Example: Keras.add.layers(etc) Hope this helps

[@riadhgharbi7985]

keep up the likes and comments lads, we need more of his content, support our guy here xD

[@lilprotakeit]

Hi Sir, Your videos are the reason why i am continuing and surviving as a data engineer. I would be grateful if you can create a series on Apache Airflow as its a heavily used framework for data engineering. Please do consider.

[@fcolecumberri]

You should add this to your keras tutorial playlist. Thanks for this and for that tutorial

[@beyhan9191]

Zero unlike! You’re doing great things

[@TheRedProject]

I started using Kite a month ago. I love it.

[@Evan_242]

This kite things looks awesome, I will definitely check it out. Thanks Harrison, hope you're doing well :)

[@Evan_242]

I download it, it's awesome ! :)

[@kacperkubicki1101]

Woah, first time my uni classes were faster to teach me something than sentdex. I might reconsider my lack of faith in their purpose ;) have you tried talos for hyperparameters optimization? We've been using it during classes and tbh it seems nicer to me than keras tuner.

[@sentdex]

Nice, I'll check out Talos.

[@mrfizzybubbs3909]

​@@sentdex It might also be worthwhile to also check out the hyperopt library.

[@Manu-jc2sx]

What optimization method is the best one? There are many, like keras tuner, Hyperopt, Talos etc..

[@GauravKumar-ch3xn]

There is a default hyper-parameter tuning available in TensorFlow, does the same thing with some pretty visualization also while attached to tensor-board, What would be interesting to see if any of these packages apply Bayesian Optimization also , that would be nicer

[@bhuvaneshs.k638]

Thanks for this.... Very helpful.... U r the guy for machine learning in python. Thqs.... !!

[@_nttai]

I'm glad I found your channel

[@fuba44]

This was great! I will go play with it right now. thank you!

[@rchuso]

I've been using Bayesian-Optimization, and this looks a lot like that.

[@meandkg]

What about cross validation? Does it support optimizing for the average score of say 5 fold cross validation? Or does it just optimize on one fold?

[@amirmasoudkiakojouri6655]

Thank you for your clear description. I have problems with kerastuning installation and import it for tuning. Would you please let me know how to install it? When I want to install kerastuner in the terminal, I see an error as below: ERROR: Could not find a version that satisfies the requirement kerastuner (from versions: none) ERROR: No matching distribution found for kerastuner

[@taylormcclenny1416]

Doing God's work, my friend!

[@ankitganeshpurkar]

Hi sir, This tutorial is simple and effective. I have query when i am applying this random search the codes runs well. But the number of layer is something else and actually layers are different in numbers. The both number don't tally most of the time. Example number of layer in a model is 7 but total layer shown is 18. What could be the problem?

[@wadyn95]

Dear Sentdex, could you introduce tensorflow object detection API? TF updated up to 2.0 and there is no fully working tutorial now... I got too many errors while trying to use that stuff

[@sentdex]

Yeah I would like to revisit the object detection stuff, but other topics keep getting in the way :D ...one day...

[@nileshmishra3796]

Awesome man, you never disappoint :)

[@moniquemarinslp]

Great stuff! Thumbs up for the tutorial and Kite (also quite cool)!

[@usamatahir7091]

I love you Harrison!

[@jorgeespinoza3938]

Pharmaceutical companies should be dreaming of having an actual physical tuner for their compounds, although I beleive the length of their testing takes bit more than just 19 seconds.

[@sankamadushan7940]

Good job Sentdex. This is great. Save lot of time.

[@interpro]

Great tutorial! Thanks much!

[@MrLiquimatter]

sold on Kite!

[@yoannrey5286]

Hello ! Thanks for the video :) One question, did you manage to use Keras-tuner with Tensorboard ?

[@programerahmed4470]

Great video: How can I force Keras Tuner to use default hyperparameter values for the first optimization iteration

[@kerolesmonsef4179]

you are great . Thank you

[@angelazhang9082]

Thanks for the thorough video. I've been trying to figure out a way to find batch_size that the tuner found the best results with, but I've been unsuccessful. Can you comment on that? I watched your video several times and don't think you mentioned anything about batch size, which is a very common parameter to test with. I looked up several articles and haven't found any information on that either. I also haven't found any information on how to add batch size as a parameter for the tuner. So the only thing I can think of is to run the tuner multiple times for the varying batch sizes, but I'm sure there's a better way.

[@felixmuller9062]

First of all thank you very much for this amazing video. Helped me a lot! I still have a quastion. Is it possible to give the Coice function a "none" as a value? I´m aiming for a HP-Optimization where I want to try different regularizers. One option shall be that I don´t use any optimizer. Is this possible with keras_tuner?

[@eranfeit]

Thank you for great video

[@mattb9823]

This is awesome. I've been learning ML for about a month, paid for a couple courses on Udemy but I seem to be learning more from your channel when trying to debug and optimize things. Quick question, is there any way to integrate TensorBoard with RandomSearch?

[@oliverpolden]

I have exactly this question. I'm just about to try but I assume you can just assign each hyperparameter to a variable and construct your Tensorboard name from those and of course remember to use the variables in your model definition. I don't see why that wouldn't work.

[@nirbhay_raghav]

I believe tensorboard has a "what-if" option. You need to provide your model with data directories. It would not exactly be a random search but it is better than nothing. Check it out , you may find it useful.

[@oliverpolden]

How does keras-tuner compare with Tensorboard's hparams? Seems hparams would be better for analysis within Tensorboard?

[@riyabanerjee2656]

I get the error "RuntimeError: Model-building function did not return a valid Keras Model instance, found ". Any idea what I should? I googled it, and this was written: "If you want to return more than one Keras Model, you'll have to override Tuner or BaseTuner. In this case, I recommend overriding BaseTuner, since Tuner assumes a single Keras Model but BaseTuner works for any arbitrary object(s). The methods you'll need to override are BaseTuner.run_trial, BaseTuner.save_model, and BaseTuner.load_model The docstring of BaseTuner.run_trial should have enough info to get you started with how to do this, if not please let me know: github.com/keras-team/keras-tuner/blob/master/kerastuner/engine/base_tuner.py#L134" I did not quite understand the error. Any idea?

[@neatpolygons8500]

oh yeah, Kite. It's fricking genius and I use it with vim

[@nano7586]

I ALWAYS wondered how there is no optimizer for hyperparameters. People working with neural networks and machine learning but talking about "trial and error" when it comes to HYPER and not HYPO parameters. This always really confused me. It's basically like applying a neural network to the neural network. Sure, it takes a long time and is CPU/GPU expensive, but if needed you can run it overnight or even for longer times. But that also overfits your model to the validation data you are using for optimization, right? Anyways, thanks so much for sharing!

[@gouki1001]

Is it a norm to use keras tuner and keras callbacks to optimise? OR these are two methods not needing to utilize each other

[@1991kushagra]

That was really an awesome video. Hats off. I have an additional doubt in this. What if we want to use cross validation also together with random search? In scikit learn we can do that by randomizedsearchCV, is there any way in Keras also?

[@MultiNarutoGamer]

@sentdex Is it possible to tell the model to try it with and without max pooling? Or with different activation functions?

[@mdashad439]

Best Python Tutorial ever very understandable.

[@ggpopa1319]

But then why don't use an optimiser like Adam or SGD to optimise the hyperparameters too?

[@joeboyle7390]

Because evaluating the function (training an entire model) is incredibly computationally expensive compared to evaluating a single epoch. Tldr its too slow and the function is probably not convex!

[@francescaalfieri5187]

Thanks for this video!!! I have a question, is there a way to check the value assumed by the variable hp.Int("inputs_unit") in every step? I have already tried to use debug with no success.

[@Yisi.voyager]

Does the keras tuner tell you how many layers is the most optimal?

[@meandkg]

so.... Keras Tuner is better than writing for loops and testing manually? Can it get stuck in local optima?

[@gianlucavernia9444]

Hey Sentdex are you going to continue the quantum programming series or is it finished?

[@marmar321]

I forgot to save the pickle file for my test. By any way, is it possible to do a load summary in a previous run of keras tuner without pickle? Thanks

[@tingyizhu3691]

R package has plot_tune function to have a nice visualization of the tuning results. Does python have similar thing?

[@andris788]

Would this work if you have a mixed input NN? I'm trying to implement this to mine. It has a CNN and an MLP combined in a final dense layer. Keras-Tuner doesn't like if I divide X_train to [X_train_cnn, X_train_mlp].

[@patrickduhirwenzivugira4729]

Thank you for the great video. How can I also tune the optimizers (let's say ['Adam, RMSprop]) with dynamic learning rates? Many tutorials keep it fixed. Thank you.

[@siddheshwarimishra8042]

Respected sir, please tell me how to use the swarm optimization technique in the pre-trained model. and please suggest me can I use multiple pre-trained networks with multiple nature-based optimization techniques for multiple inputs. please.....

[@FrostEnceladus]

How do you know when you are using too many or too few neurons? And how do you solve the number of neuron per layer from the number of layers needed. That's my problem

[@alberto.polini]

Thank you sentdex i love your videos

[@maliksalman1907]

Sir, I need to ask you about the firefly algorithm to optimize CNN model.

[@Zifox20]

Interesting feature, thanks !

[@coder3652]

Thanks for video

[@sriadityab4794]

Can you tell me how to perform cross-validation/hyper parameter tuning for time series forecasting using LSTM?

[@neoblackcyptron]

Sorry I am worse than a paper bag. I could not solve the fashion MNIST problem manually by different layers sizes and depths for a non-CNN FCN, I could not cross the 90% val_accuracy. That is why I am going to use the keras tuner.

[@deepakkumarjoshi]

Thanks for the great work, how do we plot the result to compare, actual, predicted datasets after using the tuner?

[@jumpthecagemma4987]

Also, can someone properly explain to me what .reshape(-1,28,28,1) does? I know the 28,28 re shapes the x and y sizing and the positive 1 at the end makes it all one dimensional, but am i missing something @sentdex?

[@TheMaytschi]

Great video!! @sentdex or anyone else: I am using the tuner for RNN with stacked LSTM layers, but for some reason the tuner does not converge whereas if I try the same architecture during normal fitting, it converges. Any idea why this could happen?

[@david-vr1ty]

Nice tutorial! While watching I came up with some questions regarding overfitting/generalization: 1. Does Keras-Tuner searches for the best model considering overfitting? We specify the parameters for training (epochs & batch size), so is Keras-Tuner somehow considering overfitting in the model comparison or is it just comparing the acc of each model after the specified epchos rigardingless the number of epoch leads to overfitting or not? 2. If it does not, is the tuner still usefull? 3. If it does, can we show the number of epochs used for each model in the model report? Thx in advance ;)

[@omarabobakr2292]

david I don’t know about whether or not Keras tuner does that, but callbacks in keras might help with this task. You can let your model train with a high number of epochs, but after each epoch the model will save its weights to a ckpt file in your drive. When training is done you could load the weights of each epoch to your model and evaluate your test data.

[@pushkarajpalnitkar1695]

@@omarabobakr2292 Agree but callbacks argument is only available while executing fit, predict or evaluate methods. We are not using none of these methods here. So how and where can I use earlystopping while using tuner?

[@alberro96]

How could I implement this with CNN? I'm working with my own dataset adn it seems like the keras tuners don't like the tf.data.Datasets yet. They're still expecting (x_train, y_train), (x_test, y_test). Is my thinking correct there? Essentially I'm loading my data using tf.keras.preprocessing.image_dataset_from_directory and would like to feed this into the tune. How could I split my own data in (x_train, y_train), (x_test, y_test)?

[@51nibbler]

thx for good explain

[@walisyed4625]

Very useful, thanks

[@jm10oct]

WOW!!! that might have just made my project 3 months shorter!!!!

[@jakaseptiadi1752]

I'm thinking about changing keras optimizer algorithm during training. Is it possible in keras?

[@chaitanyasharma6270]

why did you remove maxpooling , is there a way to add some maxpooling layers?

[@rogervaldivia7033]

Thanks for the video! Do you know if its possible to optimize to cross validation error?

[@edeneden97]

Is it random search or does it use some genetic algorithm / other RL stuff?

[@yazanmajzoub6582]

really..... thanks

[@pushkarajpalnitkar1695]

Graet video! Can anyone please suggest me the number of epochs to use in the search? More specifically will using more number of epochs helps the search? Or small say 1-3 epochs are sufficient for comparison of model performance?

[@spitfire-dragonboatita9610]

I have a problem, when i put "hp"into the build_model function's argument it gives an error: "NameError: name 'hp' is not defined"; I've already import keras and I've following step by step your tutorial...but it doesn't work :(

[@Gavinnnnnnnnnnnnnnn]

this is just like grid search cross validation, which has existed for years

[@sentdex]

I don't think I or anyone claimed it was a novel concept, just that many people are likely doing it either manually or by their own new code every time and there's a lib to help.

[@chaimaaessayeh8929]

Very interesting!! Is there a way to apply this same technique on a reinforcement learning model? like the one you build in another video series?

[@luispintoc]

You'd use the bayesian optimizer instead of the random search

[@matt_t937]

Hi! thank you for the quality of your videos, you are doing an awesome job! I wanted to ask you if there you know how to tune keras models hyperparameters using Sklearn TimeSeriesSplit cross validation method and not just a shuffling cross validation like in yuor model. I tried to use Sklearn tuner but it doesn't work with my deep learning model however I really really need that cv option... help me please I need to finish up my Bachelor thesis, I can pay :)

[@davidcristobal7152]

Hi sentdex, nice video. Is there any way to integrate this keras-tuner with keras-rl (reinforcement learning) and custom environments with open ai gym interface?

[@sentdex]

I am not sure, but I don't think so at this stage.

[@kaustubhkulkarni]

How do we save and checkpoint the kerastuner random models?

[@ahsanrao6164]

If someone has issues during running this code then remove this line from your model model.add(MaxPooling2D(pool_size=(2, 2))) My code was not working, and when I remove this line, it works fine now.

[@thomhomsma5474]

How do you use tensorboard after running the search?

[@rafaelstevenson]

Hello, i seems to have problem in using keras tuner that the result shows disagreement , if you understand and care to help here is the detailed issue statement in stack overflow questions/66783048/keras-tuner-uses-for-i-in-rangehp-intn-layers-1-3-but-does-not-show-agre

[@plutoboy8827]

Thank You !!!!!!!!!

[@iskrabesamrtna]

I still cant figure out how is even possible to have -1 in reshaping while creating x and y train and test labels

[@erosennin950]

I would enjoy a kaggle-challenge playlist :D what do you think?

[@sentdex]

I've thought about it a few times but Kaggle comps often have overly burdensome rules associated with their datasets...and that tends to scare me off from doing a series

[@erosennin950]

@@sentdex got ya :(

[@leonshamsschaal]

@sentdex can we have a building nn from scratch?

[@sentdex]

It's coming!

[@RojinaPanta1]

how can we carry out search on train_on_batch dataset ?

[@rezan6971]

would you please take a look at fastapi and make a tutorial, todo app with react maybe(for someone who already knows react) or at least the back end of it without frontend

[@minazulkhan8287]

hi dear i m working on tkinter . i used ur code for multiple windows using tkinter ...the code works fine but when i used inbuild function to display current time in second window it gave error " module tkinter has no attrribute time" the code line is : localtime =time.asctime(time.localtime(time.time()) the next line includes label with a term text= localtime plz give aolution soon

[@varunnagpal2258]

I ran it with varying number of layers, but it shows strange mismatch between number of layers reported and value of num_layers [Trial summary] |-Trial ID: 79cd7bb6146b4c243eb2bc51f19985de |-Score: 0.8444444537162781 |-Best step: 0 > Hyperparameters: |-Conv2D_0: 448 |-Conv2D_1: 448 |-Conv2D_2: 512 |-learning_rate: 0.0001 |-num_layers: 1 |-rate: 0.5 You can see there are three Conv2D layers and yet it shows num_layers as 1 ...why ? def build_model(hp): model = tf.keras.Sequential(); model.add(base_model); for i in range(hp.Int('num_layers', 1, 2)): model.add(tf.keras.layers.Conv2D(filters=hp.Int('Conv2D_' + str(i), min_value=32, max_value=512, step=32), kernel_size=3, activation='relu')); model.add(tf.keras.layers.Dropout(hp.Choice('rate', [0.3, 0.5]))); model.add(tf.keras.layers.GlobalAveragePooling2D()); model.add(tf.keras.layers.Flatten()); model.add(tf.keras.layers.Dropout(0.2)); model.add(tf.keras.layers.Dense(5, activation='softmax')); model.compile(optimizer=tf.keras.optimizers.RMSprop(hp.Choice('learning_rate', [1e-4, 1e-5])), loss='categorical_crossentropy', metrics=['accuracy']); return model

[@nmana9759]

Can this tuner used for RNN, Please answer thank you

[@Yourbitchiscrazy]

Can and if how do, you use tensorboard and keras tuner together?

[@jumpthecagemma4987]

What playlist will this be added to?

[@shayekhbinislam]

What is the best counterpart of keras tuner for pytorch?

[@jasonproconsult9525]

I'm using Spyder which prints tuner.results_summary() as an . Seems no workaround yet from Spyder dev :(