New year’s resolution: dive into deep learning!

This is Daniel Whitenack, a data scientist with SIL International, and you’ve joined us for another Fully Connected episode of Practical AI. In these Fully Connected episodes my co-host Chris, who’s a Chief AI strategist at Lockheed Martin, and I keep you fully connected with everything that’s happening in the AI community. We take some time to discuss the latest AI news, and dig into some learning resources to help us all level up our machine learning game.

Welcome, Chris. Good to talk to you again in the new year.

Happy new year, man! Good to talk to you, too. We’ve got some exciting stuff coming up this year.

Yeah, for sure. This is a very special episode of Fully Connected. Normally, we go through a bunch of the latest news and that sort of thing, but we’ve decided for this episode to give you our AI new year’s resolutions for the year, and go through our thought process of how we go about learning new techniques, new methodologies, new theory related to AI and deep learning. This should be pretty fun. Are you excited, Chris?

I’m really excited. I think this is a long time coming. We’re both always advocating new learning resources and stuff, and I think it’ll be fun to share what we each have and how we go about the process, because I don’t know about you, but I get asked that all the time.

Yeah. There’s so many resources out there, and really, there’s such a wide variety of resources in terms of the background that they expect people to have, what tooling they use, what languages they use, and so it can be really overwhelming for people trying to pinpoint the right way to learn new subjects and dive into new things as related to AI and machine learning.

Maybe before we jump into our specific resolutions, Chris and myself, maybe we can just talk a little bit about the thought process that we go through when we’re thinking about how to select the right sorts of resources. Now, in my mind, one of the things that I’m thinking about when I’m looking at resources - for me, there’s resources out there related to deep learning and AI and other things that are really focused on research, and then there’s a bunch of things really related to maybe the application of AI. Do you see a similar trend there, Chris?

Yeah, I do, and I have a strong bias to acknowledge… I’m very much interested in the implementation side. I’m definitely not an AI researcher in that sense. I’m the kind of person who likes to come along, I program, as we’re moving into the neural computing world, I love seeing some of these new capabilities coming out from all these different organizations, from Google to Amazon to Microsoft to you name it, there’s so many. I like to find something that suits me, and that’s the very first thing I do - find something that’s captured my interest, and figuring out where I wanna go.

It sounds like you would be an excellent host for some type of practical AI content creation.

There you go, we should start a podcast.

[03:59] [laughs] Yeah. I see the same thing. I think the first thing that maybe you want to be thinking of when you’re trying to find new learning resources as you go into this new year are really what you want, and there’s really no right answer to this question. Chris and I maybe lean more towards the application and integration and use of deep learning and AI and machine learning, but we need great AI researchers… So if you’re leaning towards wanting to go into research, we certainly need people like that, so I think you wanna be asking some of these questions, like do you want to do deep learning research, figure out new sorts of AI, new sorts of neural networks or techniques that haven’t been applied before, and maybe do that in an academic setting, working on something very narrow for longer periods of time? Or do you want to be in an industry setting or in some organization that’s actually applying techniques on a more rapid timescale?

That’s a great point, because even though I declared my bias on the implementation very much an industry focus, I can’t do what I wanna do unless brilliant people in Academia, or these days in industrial settings doing research are creating these amazing new tools, new architectures that we can go apply. So lest I go too far on the implementation side, I definitely want as many people to go into research as possible, selfishly, so that I can play with the fruits of their labors.

Yeah, it’s definitely needed. As we shift to our resolutions, I think those are gonna be much more application-based and practical, utilitarian maybe, if you wanna say that… But if you’re kind of leaning more towards the research side of things, often I get asked – as I’m doing workshops and talks and other things, I get asked “Oh, if I’m going into this field or if I wanna advance in AI, should I go and get a PhD?” and as we’ve already said on this show many times, you don’t need a Ph.D. to work in AI, but if you’re wanting to do A.I. research and that sort of thing, you might consider other education. You shouldn’t do a Ph.D. because it’s a requirement to work in AI, because it’s not. In my opinion, you should do a Ph.D. because you like research, and you wanna do research, and that’s often how that world works. That’s my opinion about that further education.

I don’t know, is there certain education that you feel is relevant here, Chris?

No, for research I agree with you. I would definitely say let that education be driven by your passion to do research, and I think in a lot of cases going with that Ph.D. is a great path, because it’s actually doing just that. But you’re not getting the Ph.D. just to do it. I don’t have a Ph.D. I know you do, but I don’t have a Ph.D. and I love this field, and I’m able to be productive in it, and lots of other people can. But at the same time, if someone’s gonna go into Academia, I think that’s probably the right path for them to take - jump into a passion-driven Ph.D. program where they can go do the thing they love to do.

Yeah, and there’s also – if you wanna end up doing AI research in industry, maybe not being a professor; like, you wanna work at Google Brian, or OpenAI, or one of these places, there are fellowships and opportunities for you to get involved. So if that’s maybe part of your new year’s resolutions for 2019, advancing to a goal of working with one of these companies, you can look up the OpenAI Fellows Program, the Google AI Residency… There’s also some interesting things from the Allen AI Institute; they have actually a pre-doctoral internship program, so that might be something. If you’re starting to go that way, maybe you can look into that as something to work on in 2019.

[07:52] But assuming that you don’t maybe want to go that route into research, as Chris and I have not pursued that route as much, but maybe you want to just learn more about deep learning, dive deeper into deep learning, I think that it’s perfectly fine to consider some self-study options, some kind of semi-guided study options… Maybe there’s bootcamps, there’s courses, there’s a bunch of hybrid material, that include code practicums and exercises and videos and text pieces… So there’s a whole lot of options out there, but maybe to start out and narrow us in on some of the ones that we’re interested in for 2019, maybe this is a good time when we can share some of our new year’s resolutions as related to AI.

I’ve applied a lot of - I don’t know if I can use the word “traditional”, but more traditional sort of machine learning techniques - maybe regressions, and decision trees, random forests, a lot of these techniques in the past on various use cases, but I think I need to dive a little bit deeper into the neural network methods. I know we’ve talked a lot about them on this show, and I know a lot of what’s out there, but I’d like to dive deeper into some of those methods.

Specifically, I think my resolution is to figure out some of the things that are going on with NLP and deep learning. I specifically wanna dive into that side of deep learning in this new year and learn a little bit more about how that works, and how I can apply it, especially since I work with an organization that’s primarily concerned with language.

Do you have any specific new year’s resolutions, or things that you would like to level up on this coming year, Chris?

Yeah, and it’s a bit different this year than, say, last year at this time, because as we have talked about, so many new tools and frameworks have come about, and the capability of how far you can make it in a certain area without having to be just purely an expert in that area - that keeps getting easier and easier to manage from an implementation standpoint… So I have a keen interest in robotics, and I’ve been in that world professionally. One of the things that I’m very passionate about, aside from strictly work-related, is doing stuff with my daughter, and so one of the things this coming year that I’m planning to do is start to take some of the lower-hanging fruits that are available on the NLP side and on the machine vision side and put them into some simple robotics things that I can share with her. That has me very excited, totally outside of work… And it’s funny, she’s already able to do that.

So if you would have asked me a year ago, it was very work-related, and at this point it’s almost kind of bringing it home, to some degree, and being able to share it in more of a day-to-day kind of sense.

Awesome. Yeah, I always love hearing about your passion to make sure that you’re both learning, but also able to contribute and integrate your family relationships into this sort of work, and with your daughter learning all these things I think it’s so cool.

With my resolution, and kind of how I went about thinking about the resources that I’m gonna target for this next year, I wanna learn about natural language processing, and I want to dive deeper into the deep learning methods as related to language.

What I did was basically google search “nlp” and “PyTorch” and “TensorFlow” and “deep learning”, and search the O’Reilly website, and search other websites, and Amazon, and all of these things, and the shortlist of what I came down to – there are a lot of things out there, but there’s a new book coming out from O’Reilly titled “NLP with PyTorch”, which seems very relevant to me, because I’ve had some experience with PyTorch in the past, and it’s relatively easier for me to understand in some respects. This is coming out soon - release date is January 2019. I’m excited… It covers apparently a bunch of things from recurrent neural networks, to other things like LSTMs, and other things. I’m excited to maybe use that as a jumping off point to learning some more NLP with PyTorch.

[12:17] There’s also a course on Udacity called “Deep learning with PyTorch.” It’s not specifically geared towards NLP, but I thought it would be maybe good; as I’m learning about NLP from the book, maybe I could dive into – maybe questions will come up around how this works in PyTorch, or why they did this with PyTorch, so… I think this course, which is free, and covers things like the intro to PyTorch, along with related things like recurrent neural networks and natural language classification, I thought that would be a good supplement.

And then finally, there’s a set of videos from Pearson that are about deep learning for natural language processing. The reason why I, in my thought process, came to these videos, was I thought, well, I don’t necessarily only want to be versed in PyTorch; I also (at least personally) have the goal of being able to work both with PyTorch and TensorFlow, depending on the situation or the company I’m working with, or whatever it is. So that one is actually more TensorFlow-based, so I’d like to kind of learn both aspects of NLP and how it might be implemented slightly different in both.

So that’s the thought process that I went through for my new year’s resolution. Does that make any sense to you, Chris?

It totally makes sense, and I’m not surprised, given the passions that you’ve expressed over time, where you’re going and what you’re interested in. That’s a fantastic set of resources to dive into.

Yeah, so I realized by confessing all of these things on our podcast that probably some people will keep me accountable on these things, so feel free to do that in our Slack team, or on LinkedIn, and see how my progress is going. Or maybe if you want to go through some of these resources as I do, let me know in Slack. You can join our Slack team at Changelog.com/community. Let me know and maybe we can form a study group.

When I got into my interest in deep learning a few years ago the resources were a lot fewer at the time in terms of choice (there’s so much choice right now) so for me, I really got started with a couple of things, one of which was - it was not when I started, but it came out not too far down the road - the Deep Learning Textbook, which is by Ian Goodfellow, Yoshua Bengio and Aaron C. Courville. Sorry about that middle one there, I’ve never gotten the right pronunciation. But it’s a great aspirational read as you’re getting into it; it’s hard, it’s truly a textbook, but it’s something I’m constantly going back to as I’m trying to understand the underlying mathematics, and how the algorithms fit together… And I’ll go to other books to pick up specific things that might be easier, by O’Reilly or Packt Publishing or others like that. If it’s some of the basic concepts, I tend to come back to here and see if my understanding is better.

So I really think it’s a good investment as kind of a metric on how your learning is coming along. If you can read the deep learning textbook and it makes sense on your basics, then you probably have it.

Great reference for anyone.

It’s definitely worth the money. I have it literally right in front of me as I’m saying this right now, and it’s always nearby. And actually, there’s another great book that I’m aware of, and since I know that you would never promote it yourself, I’m going to say it, because it is a fantastic book… Daniel authors a book which is Machine Learning With Go, and it is a really, really good book; I say that not just because I’m Daniel’s friend, but because he is a fantastic data scientist and he’s just written an excellent thing… And as the not-author of the two of us, I want to note that – you may notice that it is with Go, and that’s actually how Daniel and I met, we’re both Go programmers (called gophers in the Go community) and got to know each other there first.

I was absolutely thrilled when he wrote this book, because I actually think - as does he - that Go is a great language to be able to do data science and artificial intelligence in.

[16:16] Thanks, Chris.

Yeah… So since you would never promote it yourself, I’m wanting to point that out. So I recommend everybody go to packtpub.com, or whatever distributor - Amazon, or other - and get Machine Learning With Go by Daniel Whitenack. I think it’s a great book.

I got my start early on - my first online course was the original Coursera Machine Learning Course that Andrew Ng taught, and I got my certification, but I would say that that one’s pretty much out of date at this point. There’s a more recent Coursera one on deep learning specifically, but I think as I go into 2019 and really focused on implementation, I really am trying to merge my software engineering background with my AI interests, so that they really become seamless one.

Which was one of the trends that we saw for 2019 in our last Fully Connected, right?

Absolutely. I don’t want them to feel like almost two different parts of me technically, so I’m probably gonna dive into the machine learning with AWS that Amazon has out at this point… On that one specifically because I use AWS personally for lots of things, and have been using it since I think 2010, so a long time. It’s my go-to framework.

Then I’m also interested in PyTorch and TensorFlow, and I wanna do Google’s Machine Learning Crash Course as well, with the TensorFlow API. So that’s a keen interest to me going forward. And then I tend to supplement things with YouTube all the time. There’s so many great videos on deep learning on YouTube…

I have a friend who’s actually been on our show, Chris DeBellis, who is always telling me about the latest YouTube video in deep learning, and that’s a great place to go.

Awesome. And I think in light of explaining how we have got to these good resources for us this year, we might note that maybe – I hope that these would be good resources for everyone, but they might not be the best for everyone.

I think things to keep in mind while you’re looking for learning resources this year, as you dive deeper into neural networks or whatever it might be, I think there’s really two types of routes that you might want to go. If you’re coming from a software development background, then maybe the challenging part of deep learning and neural networks and these things for you might be the more mathy things and the theory of it… Whereas if you’re coming from kind of a scientist or academic background, maybe the challenge for you is not so much the math, but it’s the coding ability that you need to develop for some of these things. So if you’re coming from a developer standpoint, maybe you wanna pick out a resource that’s really gonna stretch you math-wise, and build up some of your skills on the math side of things… So maybe looking a little bit more into the theory of recurring neural networks, or something like that.

Whereas if you’re coming from the science or academic part, maybe that comes to you rather quickly and maybe the thing that you want to focus on more is actually going through practical coding examples and making sure that the learning resource that you choose for the year (or part of the year) actually includes some practical exercises, some projects that you can work on, through real code.

For me, that latter one is generally what I look for, but I don’t know… What’s your filter as you’re looking at resources, Chris?

[19:44] That’s a great point that you’re making, and I proably should do more of that, actually… Because I can actually relate that to a personal experience. At a former employer, our AI team really had people that were software engineers who had converted over into this, and people who were data scientists purely, straight out of school, and that’s really what they did… That dichotomy that you just pointed out, where developers may not have the math background that the data scientists had, but the data scientists sometimes were struggling with the programmatic and infrastructure issues. Each side going to what they were weaker in, and building up some skill on the other thing, that you don’t do, is a really great thing to do. Maybe I need to reconsider some of mine; since I’m coming from software engineering, I’m always trying to level up on my mathematics; maybe I need to add that into my 2019 there, so… Good point you made there.

Yeah, and whichever side of that you fall on, there’s definitely good intro resources on the math side, there’s good resources on the coding side, like Python resources and other things… So if you’re struggling to find any of those, again, reach out in our community and we’d love to help you find some of those.

Learning resources, books and courses - in my opinion, they can only get you so far. At some point you need to actually apply what you’re learning and work on kind of a side project or an interesting project of some type to actually try to apply some of what you learned, and that’s really where I feel like a lot of things sink in, and you gain a lot of knowledge.

In light of our resolutions, I thought it would be good to give some examples of what a good side-project, good learning project would be like. I know Chris has some; I’ll mention mine first, since I’m kind of interested in the language aspect of things. There was some work recently by Facebook that did machine translation in an unsupervised way, which means that they didn’t have a dictionary of “this word corresponds to this word” in each language, or parallel language data, but they had monolingual data for each language, and then they did this unsupervised machine translation technique.

I thought that was really cool, especially for people around the world that speak minority languages… So the side-project that I would love to work on as I’m going through these resources is related to that. I’d love to go – they released a GitHub repo that contains the code that they used in that, and I would love to maybe try that out on my own set of language data, or maybe even modify it to use some other word embeddings or something that we’ve talked about in past episodes… Like, the newly-released embeddings called the BERT technique.

So I think my side-project will be tinkering with some of that, and see if I can get some machine translation going for these sort of low-resource scenarios… I think that would be a lot of fun. What do you think, Chris?

A couple things in terms of getting hands-on… I think in the past I’ve mentioned that I’ve been able to touch on the robotics world in terms of AI and robotics intersecting, and I think with my employer, Lockheed Martin, there’s some pretty amazing things that I’ll be able to get involved in, and I’ll talk about some of those down the road, at the right time. But I’ll address the personal side-project things that I’m interested in.

We’re always talking about AI for good, and as part of that, listeners probably don’t know about me - the other thing that I do when I’m not talking technology is I’m really big into animal advocacy and animal rescue. It’s actually how my wife and I met, and our family is very active; that’s what we do as a family - we go out and try to save critters.

One of the things I’ve wanted to do for a long time is to be able to use the machine vision technologies that are now available to be able to apply those to large-scale maps like Google Maps and such, and be able to detect dog fighting operations, so that we can stop them. There are some fairly distinct characteristics in different types of operations… And it doesn’t necessarily have to be dog fighting, it can be other things, but with these new tools that nobody has had the opportunity to learn, I’m actually actively trying to recruit various law enforcement agencies in the United States - I’ve had a number of conversations, and some other of the animal advocacy organizations here in the U.S, to form a coalition to pool together datasets and experts in the field and then apply the technology to it. So that’s something – I’m in the early stages, but I’m actively having those conversations, so I’m really excited to see what may be possible to end some suffering of animals that’s quite horrible.

[24:31] Yeah, that’s awesome.

So that’s a big part. And then the other one, which I’ve also mentioned previously, is doing stuff with my daughter. I’m really excited about getting the stuff out of the office as the only environment, and being able to bring it home, and doing projects around the house. I have some of the normal things, like - I have a Nest Hello for the doorbell and some other things like that that are kind of IoT-oriented, but I’m really interested in doing small-scale projects that we can do as a family, and they’re actually useful; the AI stuff is not my wife’s passion, but if I can produce something that’s very useful for our family, that my daughter’s involved in, then that kind of brings it into the family environment. So that’s a big part of it - small drone and robotics-related AI projects.

Yeah, so as you’re going into this year of 2019 and you’re thinking “Oh, I wanna dive into this topic” or “I’ve found these resources that I think are relevant for me”, think about some side-projects that you’re passionate about. Like Chris said, maybe it’s something that applies to your family situation, maybe it’s something that you’re just passionate about in general, like animal advocacy, or helping minority language speakers, or whatever it is… So find something you’re passionate about and just try to get something working; I think that’s a great way to learn these subjects.

If you don’t know where to start in terms of finding a good project to work on, or data related to a certain project that you’re interested in, there’s a couple of resources that I think are relevant - and maybe you have one too, Chris. The ones that came to my mind are DrivenData and DataKind. Both of these organizations run either competitions or support projects that are utilizing AI for good in some sort of way… So I would recommend looking there; maybe that’s some inspiration for you to find some really good uses of AI to work on on the side.

Also, as I mentioned for kind of my target side-project, maybe you’ve seen something in the news, or on a previous Fully Connected episode, some kind of new, advanced type of AI, or maybe just a new result… Likely there’s a GitHub repo that reproduces some of that. Maybe one way you could start diving into that is just by trying to get the code to work, maybe subbing out the data that they used and trying out your own data… That might be a good starting point as well for you to work on the side.

What about you, Chris, as you’re kind of working on these robotics things, and the AI For Good applications…? Do you have trouble with the data side of things, or other aspects? What recommendations do you have for people?

Yeah, so myself - I never have trouble with the inspiration for a new project, because it’s really driven by what I want to do, something that I’m passionate about. The thing that I’m always trying to do is find the right datasets that can contribute, and often it’s not one dataset, it might be a combination of different datasets that taken together can give me the use case that I’m trying to achieve.

This last year Google announced Google Data Search, which I think is just fantastic… Because prior to that I’d hear about datasets, I kept bookmarks on different datasets around, but Google Data Search has revolutionized that by - you can type in the keywords of what you’re interested in, in different ways, and come up with the datasets that are publically out there. There are many, many thousands of them, far more than I could have ever bookmarked on my own… So I think it’s a must-have tool that anyone in this field is gonna be using on an ongoing basis. So Google Data Search gets a big thumbs up from me.

[28:05] For sure. Recapping what we’ve talked about so far - you might have some resolutions, like us, for this new year; use these filters that we’ve talked about, find some good resources for yourself, talk to us in the community, we’ll try to guide you to that stuff. Find some good side-projects to work on to apply what you’re learning, and then I think the last thing that is really important in terms of learning is community… And I know, Chris, you definitely agree with this; you’re involved in the Atlanta Deep Learning Meetup, and things… Do you have some inspiration for listeners as far as why they should get involved in meetups, or what benefits those might be in terms of learning?

Yeah, I do, and I would urge people – I’m the organizer, I created and organize the Atlanta Deep Learning Meetup. When I started it, which was at the very beginning of 2017, when I kind of started it saying “There’s some general machine learning stuff, but I really wanna have a deep learning conversation, and I don’t know if anyone will ever show up”, but I just kind of went for it, and suddenly there was an outpouring of people who said “This is something I’m interested in [unintelligible 00:29:12.10]”, and I think we have something like 2,200 people in the group now. Obviously, they don’t all come, but…

That’s amazing. It’s awesome.

Yeah, but it’s not uncommon for us to have 100 people show up for a particular monthly event, and we do it pretty much every month. So the outcome of that isn’t just about the presentations at the meetup. It’s great when you have great speakers, doing really interesting topics, but people often say “What about filming it?”, or whatever, and they’re almost missing what I think is probably the most important aspect, and that is getting in a room and talking to other people who share that passion and interest from other organizations, or Academia. There is so much value in those conversations and creating those relationships that no matter how good the presentation is, it’s almost incidental to forming those relationships with different people… And on a larger scale, conferences are the same way. If you’re gonna go to a data or AI conference, don’t just go to the presentations, walk up and introduce yourself. Be bold, and have conversations with as many different people as you can, because some of those will really take you places you weren’t expecting, that are wonderful. Engaging other human beings in this world of AI is about one of the best things you could possibly do for yourself.

Yeah. And you know, we’re talking about learning here - everybody at those meetups, everybody at those conferences is learning. They might know a lot about a small number of subjects, but they don’t know a lot about many subjects, and that’s true for every person. You’re wanting to learn about some things - it’s perfectly fine to ask questions, and engage in discussion, because likely some of the people are having similar questions, and you can get connected to people that already know about that thing, but maybe they don’t know about some of the things that you know about… So it is really useful.

One of the things that I wanna do this coming year, since I will hopefully be diving a little bit more into this topic of NLP, is I’ve been to a lot of meetups, a lot of conferences related to operationalizing AI and machine learning; that’s mostly been my discussion at those conferences, so I’ve gone through a lot of infrastructure-related conferences, and other things, along with machine learning conferences… But I’ve never actually participated in really a more cutting-edge research sort of conference community, like EMNLP, or one of these… So maybe it would be cool this year to at least attend, or at least try to submit something based on my learnings with NLP to one of those conferences. I think that would be a part of the AI community that I haven’t interacted with as much, but I think I could learn a lot from them, so I would be excited to get involved there.

You would do very well at that, Daniel. I think you would rock the house.

[32:09] Well, I appreciate that. It will be interesting… I’m interested and just excited about a lot that’s going on in this community, and I know you are as well. It’s exciting to see things moving so fast and have so many opportunities to dive into interesting topics. I’m just excited about learning a bunch this year.

Yeah, I think we are fortunate to be able to work in what is surely the coolest field on the planet right now. It is so fast-moving, there’s constantly new things to learn… You never get established, you never get to a point where “I finished learning”, you just wait a week or two and there’s the next thing. So if you love to learn, if you love to constantly be moving, it’s a great field to be in, either professionally, or as an amateur, either way. I definitely encourage people with the interest to dive into it.

Yeah, definitely. And you know, we have limited time on the podcast, so we’re only able to share a few of the resources that are on our mind this coming year, but we’ve got a ton of learning resources that we know about, and we’re gonna list a whole bunch of those in the show notes of this episode… So make sure and check those out, find something that you can work on this year and level up your skills. And if none of those things in the show notes make sense, and even if they do make sense, again, join our community on Slack, join our community on LinkedIn, and participate in the discussion around things you’d like to see on the show, but also maybe learning resources that you’re looking for and people can help you find them.

I’m excited about 2019, I’m looking forward to learning more.

Alright, happy new year, Daniel, and happy new year to everyone out there! We have a great year ahead.

Yeah. Bye-bye!